2024-08-19T14:04:23.361 INFO:root:teuthology version: 1.2.0 2024-08-19T14:04:23.374 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-19T14:04:23.478 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-08-18_21:00:03-rados-squid-distro-default-smithi/7859720 branch: squid description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} email: ceph-qa@ceph.com first_in_suite: false job_id: '7859720' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-08-18_21:00:03-rados-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid sha1: 78215f5a1cf5eaf321d506a66539942e12851f34 owner: scheduled_teuthology@teuthology priority: 700 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 3241 sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 sleep_before_teardown: 0 subset: 18710/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 78215f5a1cf5eaf321d506a66539942e12851f34 targets: smithi102.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDqTiLwaafgwu9tYAb8N48NqjjgZojN3lcFz8hWD+f7Gp2KzyOzLCjDDHLoJOrIqU/p4B7OWqUWbEAj+re2isbM= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 67db276914d29505c9707d04f3a3b8fb9d8aee4b timestamp: 2024-08-18_21:00:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-19T14:04:23.478 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa; will attempt to use it 2024-08-19T14:04:23.479 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa/tasks 2024-08-19T14:04:23.479 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-19T14:04:23.481 INFO:teuthology.task.internal:Checking packages... 2024-08-19T14:04:23.501 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'de09dd018de4baddfa6ca4d43bc8fb784cf75833' 2024-08-19T14:04:23.501 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-19T14:04:23.501 INFO:teuthology.packaging:ref: None 2024-08-19T14:04:23.501 INFO:teuthology.packaging:tag: None 2024-08-19T14:04:23.501 INFO:teuthology.packaging:branch: squid 2024-08-19T14:04:23.501 INFO:teuthology.packaging:sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T14:04:23.502 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-08-19T14:04:23.728 INFO:teuthology.task.internal:Found packages for ceph version 19.1.0-1386.g78215f5a 2024-08-19T14:04:23.729 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-19T14:04:23.736 INFO:teuthology.task.internal:no buildpackages task found 2024-08-19T14:04:23.736 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-19T14:04:23.885 INFO:teuthology.task.internal:Saving configuration 2024-08-19T14:04:23.893 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-19T14:04:23.911 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-19T14:04:23.944 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-18_21:00:03-rados-squid-distro-default-smithi/7859720', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-19 13:59:19.563454', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDqTiLwaafgwu9tYAb8N48NqjjgZojN3lcFz8hWD+f7Gp2KzyOzLCjDDHLoJOrIqU/p4B7OWqUWbEAj+re2isbM='} 2024-08-19T14:04:23.944 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-19T14:04:23.949 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-08-19T14:04:23.949 INFO:teuthology.run_tasks:Running task console_log... 2024-08-19T14:04:23.989 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc22dd73ac0>, signals=[15]) 2024-08-19T14:04:23.989 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-19T14:04:23.995 INFO:teuthology.task.internal:Opening connections... 2024-08-19T14:04:23.995 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2024-08-19T14:04:23.996 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T14:04:24.077 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-19T14:04:24.083 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2024-08-19T14:04:24.164 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2024-08-19T14:04:24.164 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2024-08-19T14:04:24.220 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="CentOS Stream" 2024-08-19T14:04:24.221 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="9" 2024-08-19T14:04:24.221 INFO:teuthology.orchestra.run.smithi102.stdout:ID="centos" 2024-08-19T14:04:24.221 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="rhel fedora" 2024-08-19T14:04:24.221 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="9" 2024-08-19T14:04:24.221 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el9" 2024-08-19T14:04:24.221 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-19T14:04:24.221 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31" 2024-08-19T14:04:24.221 INFO:teuthology.orchestra.run.smithi102.stdout:LOGO="fedora-logo-icon" 2024-08-19T14:04:24.221 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-19T14:04:24.221 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://centos.org/" 2024-08-19T14:04:24.221 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-19T14:04:24.222 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-19T14:04:24.222 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-19T14:04:24.222 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2024-08-19T14:04:24.252 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-19T14:04:24.259 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-19T14:04:24.264 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-19T14:04:24.265 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2024-08-19T14:04:24.281 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-19T14:04:24.286 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-19T14:04:24.287 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2024-08-19T14:04:24.407 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-19T14:04:24.447 INFO:teuthology.run_tasks:Running task kernel... 2024-08-19T14:04:24.506 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-08-19T14:04:24.506 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-08-19T14:04:24.507 DEBUG:teuthology.orchestra.run.smithi102:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-19T14:04:24.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:04:24.523 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-08-19T14:04:24.581 INFO:teuthology.orchestra.run.smithi102.stdout:5.14.0-480.el9.x86_64 2024-08-19T14:04:24.581 INFO:teuthology.task.kernel:Running kernel on smithi102: 5.14.0-480.el9.x86_64 2024-08-19T14:04:24.581 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum install -y kernel 2024-08-19T14:04:26.808 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.2 MB 00:00 2024-08-19T14:04:31.241 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 6.0 MB/s | 20 MB 00:03 2024-08-19T14:04:36.784 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.6 MB 00:01 2024-08-19T14:04:38.318 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 92 kB/s | 18 kB 00:00 2024-08-19T14:04:39.447 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 22 MB/s | 22 MB 00:01 2024-08-19T14:04:44.786 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-08-19T14:04:46.701 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-19T14:04:46.701 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-5.14.0-480.el9.x86_64 is already installed. 2024-08-19T14:04:46.777 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:04:46.778 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-08-19T14:04:46.778 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:04:46.874 DEBUG:teuthology.orchestra.run.smithi102:> echo no | sudo yum reinstall kernel || true 2024-08-19T14:04:47.386 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:03 ago on Mon 19 Aug 2024 02:04:44 PM UTC. 2024-08-19T14:04:47.533 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:04:47.544 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:04:47.544 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-08-19T14:04:47.544 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:04:47.544 INFO:teuthology.orchestra.run.smithi102.stdout:Reinstalling: 2024-08-19T14:04:47.544 INFO:teuthology.orchestra.run.smithi102.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-19T14:04:47.544 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:04:47.544 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-08-19T14:04:47.544 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:04:47.544 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:04:47.545 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 1.5 M 2024-08-19T14:04:47.545 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 0 2024-08-19T14:04:47.545 INFO:teuthology.orchestra.run.smithi102.stderr:Operation aborted. 2024-08-19T14:04:47.609 INFO:teuthology.orchestra.run.smithi102.stdout:Is this ok [y/N]: 2024-08-19T14:04:47.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum reinstall -y kernel || true 2024-08-19T14:04:48.130 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:04 ago on Mon 19 Aug 2024 02:04:44 PM UTC. 2024-08-19T14:04:48.252 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:04:48.252 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:04:48.253 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-08-19T14:04:48.253 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:04:48.253 INFO:teuthology.orchestra.run.smithi102.stdout:Reinstalling: 2024-08-19T14:04:48.253 INFO:teuthology.orchestra.run.smithi102.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-19T14:04:48.253 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:04:48.253 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-08-19T14:04:48.253 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:04:48.253 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:04:48.253 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 1.5 M 2024-08-19T14:04:48.253 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 0 2024-08-19T14:04:48.253 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-08-19T14:04:48.588 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-5.14.0-480.el9.x86_64.rpm 5.2 MB/s | 1.5 MB 00:00 2024-08-19T14:04:48.588 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:04:48.589 INFO:teuthology.orchestra.run.smithi102.stdout:Total 4.5 MB/s | 1.5 MB 00:00 2024-08-19T14:04:48.625 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-08-19T14:04:48.867 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-08-19T14:04:48.868 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-08-19T14:04:48.892 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-08-19T14:04:48.892 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-08-19T14:04:49.052 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-08-19T14:04:49.140 INFO:teuthology.orchestra.run.smithi102.stdout: Reinstalling : kernel-5.14.0-480.el9.x86_64 1/2 2024-08-19T14:04:49.140 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T14:04:49.941 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T14:04:49.941 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/2 2024-08-19T14:04:50.719 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T14:04:50.720 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:04:50.720 INFO:teuthology.orchestra.run.smithi102.stdout:Reinstalled: 2024-08-19T14:04:50.720 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-19T14:04:50.720 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:04:50.720 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:04:50.910 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel | sort -rV | head -n 1 2024-08-19T14:04:50.967 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-19T14:04:50.967 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-19T14:04:50.968 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-08-19T14:04:50.968 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-08-19T14:04:50.968 DEBUG:teuthology.orchestra.run.smithi102:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-19T14:04:51.002 INFO:teuthology.orchestra.run.smithi102.stdout:ttyS1 2024-08-19T14:04:51.024 DEBUG:teuthology.parallel:result is None 2024-08-19T14:04:51.024 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-19T14:04:51.031 INFO:teuthology.task.internal:Creating test directory... 2024-08-19T14:04:51.031 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-19T14:04:51.079 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-19T14:04:51.086 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-19T14:04:51.091 INFO:teuthology.task.internal:Creating archive directory... 2024-08-19T14:04:51.091 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-19T14:04:51.149 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-19T14:04:51.155 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-19T14:04:51.155 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-19T14:04:51.242 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:04:51.252 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:04:51.254 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-19T14:04:51.260 INFO:teuthology.task.internal:Configuring sudo... 2024-08-19T14:04:51.261 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-19T14:04:51.286 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-19T14:04:51.297 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-19T14:04:51.298 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-19T14:04:51.342 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T14:04:51.437 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T14:04:51.478 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T14:04:51.568 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T14:04:51.601 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-08-19T14:04:51.601 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-19T14:04:51.665 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2024-08-19T14:04:51.746 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T14:04:52.070 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-19T14:04:52.077 INFO:teuthology.task.internal:Starting timer... 2024-08-19T14:04:52.077 INFO:teuthology.run_tasks:Running task pcp... 2024-08-19T14:04:52.089 INFO:teuthology.run_tasks:Running task selinux... 2024-08-19T14:04:52.102 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-08-19T14:04:52.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo service auditd rotate 2024-08-19T14:04:52.166 INFO:teuthology.orchestra.run.smithi102.stdout:Rotating logs: 2024-08-19T14:04:52.169 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-19T14:04:52.169 DEBUG:teuthology.orchestra.run.smithi102:> /usr/sbin/getenforce 2024-08-19T14:04:52.188 INFO:teuthology.orchestra.run.smithi102.stdout:Permissive 2024-08-19T14:04:52.188 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi102.front.sepia.ceph.com': 'permissive'} 2024-08-19T14:04:52.188 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-19T14:04:52.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:04:52.254 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-19T14:04:52.254 DEBUG:teuthology.orchestra.run.smithi102:> sudo /usr/sbin/setenforce permissive 2024-08-19T14:04:52.321 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-19T14:04:52.330 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-19T14:04:52.346 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-19T14:04:52.347 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi102.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-19T14:06:34.476 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi102.front.sepia.ceph.com')] 2024-08-19T14:06:34.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-08-19T14:06:34.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T14:06:34.606 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-08-19T14:06:34.690 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-08-19T14:06:34.690 INFO:teuthology.run_tasks:Running task clock... 2024-08-19T14:06:34.700 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-19T14:06:34.700 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-19T14:06:34.701 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T14:06:34.758 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-19T14:06:34.773 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-19T14:06:34.797 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: ntpd: command not found 2024-08-19T14:06:34.808 INFO:teuthology.orchestra.run.smithi102.stdout:506 Cannot talk to daemon 2024-08-19T14:06:34.822 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-19T14:06:34.836 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-19T14:06:34.891 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-08-19T14:06:34.894 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T14:06:34.895 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-08-19T14:06:34.895 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T14:06:34.895 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T14:06:34.895 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T14:06:34.895 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T14:06:34.896 INFO:teuthology.run_tasks:Running task install... 2024-08-19T14:06:34.903 DEBUG:teuthology.task.install:project ceph 2024-08-19T14:06:34.904 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'de09dd018de4baddfa6ca4d43bc8fb784cf75833'}} 2024-08-19T14:06:34.904 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'de09dd018de4baddfa6ca4d43bc8fb784cf75833'} 2024-08-19T14:06:34.904 INFO:teuthology.task.install:Using flavor: default 2024-08-19T14:06:34.909 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-19T14:06:34.909 INFO:teuthology.task.install:extra packages: [] 2024-08-19T14:06:34.910 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'de09dd018de4baddfa6ca4d43bc8fb784cf75833', 'tag': None, 'wait_for_package': False} 2024-08-19T14:06:34.910 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T14:06:35.073 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/ 2024-08-19T14:06:35.073 INFO:teuthology.task.install.rpm:Package version is 19.1.0-1382.gde09dd01 2024-08-19T14:06:35.192 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-19T14:06:35.193 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-08-19T14:06:35.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-19T14:06:35.218 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-19T14:06:35.218 DEBUG:teuthology.orchestra.run.smithi102:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/de09dd018de4baddfa6ca4d43bc8fb784cf75833/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-19T14:06:35.285 DEBUG:teuthology.orchestra.run.smithi102:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-19T14:06:35.361 DEBUG:teuthology.orchestra.run.smithi102:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-19T14:06:35.426 INFO:teuthology.orchestra.run.smithi102.stdout:check_obsoletes = 1 2024-08-19T14:06:35.427 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum clean all 2024-08-19T14:06:35.651 INFO:teuthology.orchestra.run.smithi102.stdout:45 files removed 2024-08-19T14:06:35.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-radosgw 2024-08-19T14:06:36.211 INFO:teuthology.orchestra.run.smithi102.stdout:ceph packages for x86_64 318 kB/s | 84 kB 00:00 2024-08-19T14:06:36.495 INFO:teuthology.orchestra.run.smithi102.stdout:ceph noarch packages 48 kB/s | 12 kB 00:00 2024-08-19T14:06:36.770 INFO:teuthology.orchestra.run.smithi102.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-08-19T14:06:38.562 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.2 MB 00:01 2024-08-19T14:06:41.697 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 9.8 MB/s | 20 MB 00:02 2024-08-19T14:06:47.406 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.6 MB 00:01 2024-08-19T14:06:48.890 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 18 kB 00:00 2024-08-19T14:06:50.061 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 21 MB/s | 22 MB 00:01 2024-08-19T14:06:55.467 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-08-19T14:06:57.694 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:06:57.695 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:06:57.696 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-08-19T14:06:57.696 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:06:57.696 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-08-19T14:06:57.696 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 11 M 2024-08-19T14:06:57.696 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrading: 2024-08-19T14:06:57.697 INFO:teuthology.orchestra.run.smithi102.stdout: librados2 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 3.4 M 2024-08-19T14:06:57.697 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 3.2 M 2024-08-19T14:06:57.697 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-08-19T14:06:57.697 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-19T14:06:57.697 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 5.4 M 2024-08-19T14:06:57.698 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 21 M 2024-08-19T14:06:57.698 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 25 k 2024-08-19T14:06:57.698 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-19T14:06:57.698 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-19T14:06:57.698 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-19T14:06:57.699 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 753 k 2024-08-19T14:06:57.699 INFO:teuthology.orchestra.run.smithi102.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-19T14:06:57.699 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-19T14:06:57.699 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-19T14:06:57.699 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 507 k 2024-08-19T14:06:57.700 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-19T14:06:57.700 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 5.4 M 2024-08-19T14:06:57.700 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-19T14:06:57.700 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-19T14:06:57.700 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-19T14:06:57.701 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 45 k 2024-08-19T14:06:57.701 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 136 k 2024-08-19T14:06:57.701 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 188 k 2024-08-19T14:06:57.701 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-19T14:06:57.702 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 369 k 2024-08-19T14:06:57.702 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 350 k 2024-08-19T14:06:57.702 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 117 k 2024-08-19T14:06:57.702 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-19T14:06:57.702 INFO:teuthology.orchestra.run.smithi102.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-19T14:06:57.703 INFO:teuthology.orchestra.run.smithi102.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-19T14:06:57.703 INFO:teuthology.orchestra.run.smithi102.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-19T14:06:57.703 INFO:teuthology.orchestra.run.smithi102.stdout:Installing weak dependencies: 2024-08-19T14:06:57.703 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-19T14:06:57.703 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:06:57.704 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-08-19T14:06:57.704 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:06:57.704 INFO:teuthology.orchestra.run.smithi102.stdout:Install 30 Packages 2024-08-19T14:06:57.704 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrade 2 Packages 2024-08-19T14:06:57.704 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:06:57.705 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 61 M 2024-08-19T14:06:57.705 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-08-19T14:06:58.701 INFO:teuthology.orchestra.run.smithi102.stdout:(1/32): ceph-base-19.1.0-1382.gde09dd01.el9.x86 8.9 MB/s | 5.4 MB 00:00 2024-08-19T14:06:58.769 INFO:teuthology.orchestra.run.smithi102.stdout:(2/32): ceph-selinux-19.1.0-1382.gde09dd01.el9. 378 kB/s | 25 kB 00:00 2024-08-19T14:06:58.860 INFO:teuthology.orchestra.run.smithi102.stdout:(3/32): libcephfs2-19.1.0-1382.gde09dd01.el9.x8 8.0 MB/s | 753 kB 00:00 2024-08-19T14:06:58.994 INFO:teuthology.orchestra.run.smithi102.stdout:(4/32): ceph-radosgw-19.1.0-1382.gde09dd01.el9. 12 MB/s | 11 MB 00:00 2024-08-19T14:06:59.019 INFO:teuthology.orchestra.run.smithi102.stdout:(5/32): libradosstriper1-19.1.0-1382.gde09dd01. 3.1 MB/s | 507 kB 00:00 2024-08-19T14:06:59.078 INFO:teuthology.orchestra.run.smithi102.stdout:(6/32): python3-ceph-argparse-19.1.0-1382.gde09 771 kB/s | 45 kB 00:00 2024-08-19T14:06:59.278 INFO:teuthology.orchestra.run.smithi102.stdout:(7/32): ceph-common-19.1.0-1382.gde09dd01.el9.x 18 MB/s | 21 MB 00:01 2024-08-19T14:06:59.303 INFO:teuthology.orchestra.run.smithi102.stdout:(8/32): python3-ceph-common-19.1.0-1382.gde09dd 603 kB/s | 136 kB 00:00 2024-08-19T14:06:59.320 INFO:teuthology.orchestra.run.smithi102.stdout:(9/32): python3-cephfs-19.1.0-1382.gde09dd01.el 4.4 MB/s | 188 kB 00:00 2024-08-19T14:06:59.362 INFO:teuthology.orchestra.run.smithi102.stdout:(10/32): python3-rados-19.1.0-1382.gde09dd01.el 6.2 MB/s | 369 kB 00:00 2024-08-19T14:06:59.387 INFO:teuthology.orchestra.run.smithi102.stdout:(11/32): python3-rbd-19.1.0-1382.gde09dd01.el9. 5.1 MB/s | 350 kB 00:00 2024-08-19T14:06:59.412 INFO:teuthology.orchestra.run.smithi102.stdout:(12/32): python3-rgw-19.1.0-1382.gde09dd01.el9. 2.3 MB/s | 117 kB 00:00 2024-08-19T14:06:59.496 INFO:teuthology.orchestra.run.smithi102.stdout:(13/32): librgw2-19.1.0-1382.gde09dd01.el9.x86_ 11 MB/s | 5.4 MB 00:00 2024-08-19T14:06:59.563 INFO:teuthology.orchestra.run.smithi102.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 609 kB/s | 107 kB 00:00 2024-08-19T14:06:59.605 INFO:teuthology.orchestra.run.smithi102.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 416 kB/s | 45 kB 00:00 2024-08-19T14:06:59.622 INFO:teuthology.orchestra.run.smithi102.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 765 kB/s | 160 kB 00:00 2024-08-19T14:06:59.672 INFO:teuthology.orchestra.run.smithi102.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 845 kB/s | 42 kB 00:00 2024-08-19T14:06:59.739 INFO:teuthology.orchestra.run.smithi102.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-08-19T14:06:59.781 INFO:teuthology.orchestra.run.smithi102.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.0 MB/s | 662 kB 00:00 2024-08-19T14:06:59.807 INFO:teuthology.orchestra.run.smithi102.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-08-19T14:06:59.823 INFO:teuthology.orchestra.run.smithi102.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 428 kB/s | 36 kB 00:00 2024-08-19T14:06:59.840 INFO:teuthology.orchestra.run.smithi102.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 807 kB/s | 47 kB 00:00 2024-08-19T14:06:59.899 INFO:teuthology.orchestra.run.smithi102.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 4.0 MB/s | 309 kB 00:00 2024-08-19T14:06:59.916 INFO:teuthology.orchestra.run.smithi102.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-08-19T14:06:59.933 INFO:teuthology.orchestra.run.smithi102.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-08-19T14:06:59.958 INFO:teuthology.orchestra.run.smithi102.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-08-19T14:06:59.992 INFO:teuthology.orchestra.run.smithi102.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 5.6 MB/s | 192 kB 00:00 2024-08-19T14:07:00.025 INFO:teuthology.orchestra.run.smithi102.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-08-19T14:07:00.100 INFO:teuthology.orchestra.run.smithi102.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-08-19T14:07:00.209 INFO:teuthology.orchestra.run.smithi102.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-08-19T14:07:00.276 INFO:teuthology.orchestra.run.smithi102.stdout:(31/32): librados2-19.1.0-1382.gde09dd01.el9.x8 14 MB/s | 3.4 MB 00:00 2024-08-19T14:07:00.360 INFO:teuthology.orchestra.run.smithi102.stdout:(32/32): librbd1-19.1.0-1382.gde09dd01.el9.x86_ 12 MB/s | 3.2 MB 00:00 2024-08-19T14:07:00.364 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:07:00.364 INFO:teuthology.orchestra.run.smithi102.stdout:Total 23 MB/s | 61 MB 00:02 2024-08-19T14:07:00.466 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-08-19T14:07:00.663 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-08-19T14:07:00.663 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-08-19T14:07:01.136 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-08-19T14:07:01.137 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-08-19T14:07:02.585 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-08-19T14:07:02.660 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-08-19T14:07:02.697 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-08-19T14:07:02.722 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-08-19T14:07:02.746 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-19T14:07:02.773 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-19T14:07:03.076 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-19T14:07:03.134 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-19T14:07:03.462 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-08-19T14:07:03.600 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librados2-2:19.1.0-1382.gde09dd01.el9.x86_64 7/34 2024-08-19T14:07:03.697 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:19.1.0-1382.gde09dd01.el9.x86_64 7/34 2024-08-19T14:07:03.788 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rados-2:19.1.0-1382.gde09dd01.el9.x86_64 8/34 2024-08-19T14:07:03.818 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs2-2:19.1.0-1382.gde09dd01.el9.x86_64 9/34 2024-08-19T14:07:03.881 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs2-2:19.1.0-1382.gde09dd01.el9.x86_64 9/34 2024-08-19T14:07:03.910 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-08-19T14:07:03.955 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-argparse-2:19.1.0-1382.gde09dd01.el9. 11/34 2024-08-19T14:07:04.030 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cephfs-2:19.1.0-1382.gde09dd01.el9.x86_64 12/34 2024-08-19T14:07:04.054 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libradosstriper1-2:19.1.0-1382.gde09dd01.el9.x86_6 13/34 2024-08-19T14:07:04.109 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libradosstriper1-2:19.1.0-1382.gde09dd01.el9.x86_6 13/34 2024-08-19T14:07:04.143 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-08-19T14:07:04.251 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-08-19T14:07:04.286 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-08-19T14:07:04.326 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-08-19T14:07:04.352 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-08-19T14:07:04.381 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-08-19T14:07:04.414 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-08-19T14:07:04.724 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-08-19T14:07:04.768 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librbd1-2:19.1.0-1382.gde09dd01.el9.x86_64 22/34 2024-08-19T14:07:04.871 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:19.1.0-1382.gde09dd01.el9.x86_64 22/34 2024-08-19T14:07:04.897 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rbd-2:19.1.0-1382.gde09dd01.el9.x86_64 23/34 2024-08-19T14:07:04.933 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-08-19T14:07:05.406 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-08-19T14:07:05.437 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librgw2-2:19.1.0-1382.gde09dd01.el9.x86_64 26/34 2024-08-19T14:07:05.501 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librgw2-2:19.1.0-1382.gde09dd01.el9.x86_64 26/34 2024-08-19T14:07:05.576 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rgw-2:19.1.0-1382.gde09dd01.el9.x86_64 27/34 2024-08-19T14:07:05.608 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-common-2:19.1.0-1382.gde09dd01.el9.x8 28/34 2024-08-19T14:07:07.902 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 29/34 2024-08-19T14:07:07.930 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 29/34 2024-08-19T14:07:08.477 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 29/34 2024-08-19T14:07:08.508 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-base-2:19.1.0-1382.gde09dd01.el9.x86_64 30/34 2024-08-19T14:07:08.597 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-base-2:19.1.0-1382.gde09dd01.el9.x86_64 30/34 2024-08-19T14:07:08.597 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-19T14:07:08.597 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-19T14:07:08.597 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:08.626 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-selinux-2:19.1.0-1382.gde09dd01.el9.x86_64 31/34 2024-08-19T14:07:18.268 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-selinux-2:19.1.0-1382.gde09dd01.el9.x86_64 31/34 2024-08-19T14:07:18.268 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /sys 2024-08-19T14:07:18.268 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /proc 2024-08-19T14:07:18.268 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /mnt 2024-08-19T14:07:18.268 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /var/tmp 2024-08-19T14:07:18.269 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /home 2024-08-19T14:07:18.269 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /root 2024-08-19T14:07:18.269 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /tmp 2024-08-19T14:07:18.269 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:19.244 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-radosgw-2:19.1.0-1382.gde09dd01.el9.x86_64 32/34 2024-08-19T14:07:19.270 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-radosgw-2:19.1.0-1382.gde09dd01.el9.x86_64 32/34 2024-08-19T14:07:19.270 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:07:19.270 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-19T14:07:19.270 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T14:07:19.270 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T14:07:19.270 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:19.271 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-19T14:07:19.355 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-19T14:07:19.355 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-19T14:07:20.589 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-19T14:07:20.589 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-base-2:19.1.0-1382.gde09dd01.el9.x86_64 1/34 2024-08-19T14:07:20.589 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 2/34 2024-08-19T14:07:20.589 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-radosgw-2:19.1.0-1382.gde09dd01.el9.x86_64 3/34 2024-08-19T14:07:20.589 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-selinux-2:19.1.0-1382.gde09dd01.el9.x86_64 4/34 2024-08-19T14:07:20.589 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs2-2:19.1.0-1382.gde09dd01.el9.x86_64 5/34 2024-08-19T14:07:20.589 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libradosstriper1-2:19.1.0-1382.gde09dd01.el9.x86_6 6/34 2024-08-19T14:07:20.589 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librgw2-2:19.1.0-1382.gde09dd01.el9.x86_64 7/34 2024-08-19T14:07:20.589 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-argparse-2:19.1.0-1382.gde09dd01.el9. 8/34 2024-08-19T14:07:20.589 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-common-2:19.1.0-1382.gde09dd01.el9.x8 9/34 2024-08-19T14:07:20.590 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cephfs-2:19.1.0-1382.gde09dd01.el9.x86_64 10/34 2024-08-19T14:07:20.590 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rados-2:19.1.0-1382.gde09dd01.el9.x86_64 11/34 2024-08-19T14:07:20.590 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rbd-2:19.1.0-1382.gde09dd01.el9.x86_64 12/34 2024-08-19T14:07:20.590 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rgw-2:19.1.0-1382.gde09dd01.el9.x86_64 13/34 2024-08-19T14:07:20.590 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-08-19T14:07:20.590 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-08-19T14:07:20.590 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-08-19T14:07:20.590 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-08-19T14:07:20.590 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-08-19T14:07:20.590 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-08-19T14:07:20.590 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-08-19T14:07:20.591 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-08-19T14:07:20.591 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-08-19T14:07:20.591 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-08-19T14:07:20.591 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-08-19T14:07:20.591 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-08-19T14:07:20.591 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-08-19T14:07:20.591 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-08-19T14:07:20.591 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-08-19T14:07:20.591 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-08-19T14:07:20.591 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-08-19T14:07:20.591 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:19.1.0-1382.gde09dd01.el9.x86_64 31/34 2024-08-19T14:07:20.592 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-08-19T14:07:20.592 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:19.1.0-1382.gde09dd01.el9.x86_64 33/34 2024-08-19T14:07:21.480 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-08-19T14:07:21.480 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:21.480 INFO:teuthology.orchestra.run.smithi102.stdout:Upgraded: 2024-08-19T14:07:21.480 INFO:teuthology.orchestra.run.smithi102.stdout: librados2-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:21.480 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:21.480 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-08-19T14:07:21.481 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-19T14:07:21.481 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:21.481 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:21.481 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:21.481 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:21.481 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-19T14:07:21.481 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-19T14:07:21.481 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-19T14:07:21.481 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:21.481 INFO:teuthology.orchestra.run.smithi102.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-19T14:07:21.481 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-19T14:07:21.481 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-19T14:07:21.481 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:21.482 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-19T14:07:21.482 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:21.482 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-19T14:07:21.482 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-19T14:07:21.482 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-19T14:07:21.482 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:21.482 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:21.482 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:21.482 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-19T14:07:21.482 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:21.482 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:21.482 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:21.483 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-19T14:07:21.483 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-19T14:07:21.483 INFO:teuthology.orchestra.run.smithi102.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-19T14:07:21.483 INFO:teuthology.orchestra.run.smithi102.stdout: re2-1:20211101-3.el9.x86_64 2024-08-19T14:07:21.483 INFO:teuthology.orchestra.run.smithi102.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-19T14:07:21.483 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:21.483 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:07:21.836 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-test 2024-08-19T14:07:22.389 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:27 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:07:22.521 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:07:22.522 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:07:22.522 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-08-19T14:07:22.522 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:07:22.522 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-08-19T14:07:22.522 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 49 M 2024-08-19T14:07:22.522 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-08-19T14:07:22.523 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 163 k 2024-08-19T14:07:22.523 INFO:teuthology.orchestra.run.smithi102.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-19T14:07:22.523 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-19T14:07:22.523 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:22.523 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-08-19T14:07:22.523 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:07:22.523 INFO:teuthology.orchestra.run.smithi102.stdout:Install 4 Packages 2024-08-19T14:07:22.523 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:22.524 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 49 M 2024-08-19T14:07:22.524 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 209 M 2024-08-19T14:07:22.524 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-08-19T14:07:22.932 INFO:teuthology.orchestra.run.smithi102.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.0 MB/s | 304 kB 00:00 2024-08-19T14:07:22.983 INFO:teuthology.orchestra.run.smithi102.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-08-19T14:07:23.042 INFO:teuthology.orchestra.run.smithi102.stdout:(3/4): libcephsqlite-19.1.0-1382.gde09dd01.el9. 629 kB/s | 163 kB 00:00 2024-08-19T14:07:24.336 INFO:teuthology.orchestra.run.smithi102.stdout:(4/4): ceph-test-19.1.0-1382.gde09dd01.el9.x86_ 31 MB/s | 49 MB 00:01 2024-08-19T14:07:24.337 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:07:24.338 INFO:teuthology.orchestra.run.smithi102.stdout:Total 27 MB/s | 49 MB 00:01 2024-08-19T14:07:24.365 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-08-19T14:07:24.384 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-08-19T14:07:24.384 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-08-19T14:07:24.759 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-08-19T14:07:24.760 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-08-19T14:07:25.301 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-08-19T14:07:25.369 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-19T14:07:25.409 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-19T14:07:25.445 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephsqlite-2:19.1.0-1382.gde09dd01.el9.x86_64 3/4 2024-08-19T14:07:30.011 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephsqlite-2:19.1.0-1382.gde09dd01.el9.x86_64 3/4 2024-08-19T14:07:30.086 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-test-2:19.1.0-1382.gde09dd01.el9.x86_64 4/4 2024-08-19T14:07:32.496 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-test-2:19.1.0-1382.gde09dd01.el9.x86_64 4/4 2024-08-19T14:07:32.496 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-test-2:19.1.0-1382.gde09dd01.el9.x86_64 1/4 2024-08-19T14:07:32.496 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephsqlite-2:19.1.0-1382.gde09dd01.el9.x86_64 2/4 2024-08-19T14:07:32.496 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-19T14:07:32.779 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-19T14:07:32.780 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:32.780 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-08-19T14:07:32.780 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:32.780 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:32.780 INFO:teuthology.orchestra.run.smithi102.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-19T14:07:32.780 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-19T14:07:32.780 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:32.780 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:07:33.066 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph 2024-08-19T14:07:33.627 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:38 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:07:33.771 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:07:33.773 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:07:33.773 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-08-19T14:07:33.774 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:07:33.774 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-08-19T14:07:33.774 INFO:teuthology.orchestra.run.smithi102.stdout: ceph x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 6.6 k 2024-08-19T14:07:33.774 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-08-19T14:07:33.774 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 2.4 M 2024-08-19T14:07:33.774 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 1.5 M 2024-08-19T14:07:33.774 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 249 k 2024-08-19T14:07:33.774 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 4.7 M 2024-08-19T14:07:33.774 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 17 M 2024-08-19T14:07:33.774 INFO:teuthology.orchestra.run.smithi102.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-19T14:07:33.774 INFO:teuthology.orchestra.run.smithi102.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-19T14:07:33.775 INFO:teuthology.orchestra.run.smithi102.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-19T14:07:33.775 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-19T14:07:33.775 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-19T14:07:33.775 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-19T14:07:33.775 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-19T14:07:33.775 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-19T14:07:33.775 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-19T14:07:33.775 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-19T14:07:33.775 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections 2024-08-19T14:07:33.775 INFO:teuthology.orchestra.run.smithi102.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-19T14:07:33.775 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-19T14:07:33.775 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-19T14:07:33.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-19T14:07:33.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-19T14:07:33.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-19T14:07:33.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-19T14:07:33.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-19T14:07:33.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-19T14:07:33.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-19T14:07:33.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-19T14:07:33.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-19T14:07:33.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-19T14:07:33.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-19T14:07:33.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-19T14:07:33.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-19T14:07:33.777 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-19T14:07:33.777 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-19T14:07:33.777 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-19T14:07:33.777 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-19T14:07:33.777 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-19T14:07:33.777 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:33.777 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-08-19T14:07:33.777 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:07:33.777 INFO:teuthology.orchestra.run.smithi102.stdout:Install 37 Packages 2024-08-19T14:07:33.777 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:33.778 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 30 M 2024-08-19T14:07:33.778 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 108 M 2024-08-19T14:07:33.778 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-08-19T14:07:34.530 INFO:teuthology.orchestra.run.smithi102.stdout:(1/37): ceph-19.1.0-1382.gde09dd01.el9.x86_64.r 50 kB/s | 6.6 kB 00:00 2024-08-19T14:07:34.738 INFO:teuthology.orchestra.run.smithi102.stdout:(2/37): ceph-mgr-19.1.0-1382.gde09dd01.el9.x86_ 4.5 MB/s | 1.5 MB 00:00 2024-08-19T14:07:34.780 INFO:teuthology.orchestra.run.smithi102.stdout:(3/37): ceph-mds-19.1.0-1382.gde09dd01.el9.x86_ 6.2 MB/s | 2.4 MB 00:00 2024-08-19T14:07:34.873 INFO:teuthology.orchestra.run.smithi102.stdout:(4/37): ceph-mon-19.1.0-1382.gde09dd01.el9.x86_ 14 MB/s | 4.7 MB 00:00 2024-08-19T14:07:34.898 INFO:teuthology.orchestra.run.smithi102.stdout:(5/37): ceph-mgr-modules-core-19.1.0-1382.gde09 2.1 MB/s | 249 kB 00:00 2024-08-19T14:07:35.273 INFO:teuthology.orchestra.run.smithi102.stdout:(6/37): ceph-osd-19.1.0-1382.gde09dd01.el9.x86_ 31 MB/s | 17 MB 00:00 2024-08-19T14:07:35.740 INFO:teuthology.orchestra.run.smithi102.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 292 kB/s | 253 kB 00:00 2024-08-19T14:07:35.941 INFO:teuthology.orchestra.run.smithi102.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 162 kB/s | 106 kB 00:00 2024-08-19T14:07:36.041 INFO:teuthology.orchestra.run.smithi102.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 1.1 MB/s | 1.2 MB 00:01 2024-08-19T14:07:36.441 INFO:teuthology.orchestra.run.smithi102.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 193 kB/s | 135 kB 00:00 2024-08-19T14:07:36.591 INFO:teuthology.orchestra.run.smithi102.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-08-19T14:07:36.633 INFO:teuthology.orchestra.run.smithi102.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-08-19T14:07:36.667 INFO:teuthology.orchestra.run.smithi102.stdout:(13/37): python3-requests-2.25.1-8.el9.noarch.r 172 kB/s | 125 kB 00:00 2024-08-19T14:07:36.692 INFO:teuthology.orchestra.run.smithi102.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 593 kB/s | 35 kB 00:00 2024-08-19T14:07:36.734 INFO:teuthology.orchestra.run.smithi102.stdout:(15/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-08-19T14:07:36.776 INFO:teuthology.orchestra.run.smithi102.stdout:(16/37): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-08-19T14:07:36.802 INFO:teuthology.orchestra.run.smithi102.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 576 kB/s | 77 kB 00:00 2024-08-19T14:07:36.826 INFO:teuthology.orchestra.run.smithi102.stdout:(18/37): python3-urllib3-1.26.5-6.el9.noarch.rp 274 kB/s | 215 kB 00:00 2024-08-19T14:07:36.877 INFO:teuthology.orchestra.run.smithi102.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-08-19T14:07:36.902 INFO:teuthology.orchestra.run.smithi102.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 638 kB/s | 43 kB 00:00 2024-08-19T14:07:36.919 INFO:teuthology.orchestra.run.smithi102.stdout:(21/37): lua-devel-5.4.4-4.el9.x86_64.rpm 156 kB/s | 22 kB 00:00 2024-08-19T14:07:36.953 INFO:teuthology.orchestra.run.smithi102.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-08-19T14:07:36.978 INFO:teuthology.orchestra.run.smithi102.stdout:(23/37): python3-cherrypy-18.6.1-2.el9.noarch.r 4.6 MB/s | 358 kB 00:00 2024-08-19T14:07:36.995 INFO:teuthology.orchestra.run.smithi102.stdout:(24/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-08-19T14:07:37.012 INFO:teuthology.orchestra.run.smithi102.stdout:(25/37): python3-jaraco-classes-3.2.1-5.el9.noa 309 kB/s | 18 kB 00:00 2024-08-19T14:07:37.028 INFO:teuthology.orchestra.run.smithi102.stdout:(26/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-08-19T14:07:37.045 INFO:teuthology.orchestra.run.smithi102.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-08-19T14:07:37.062 INFO:teuthology.orchestra.run.smithi102.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-08-19T14:07:37.079 INFO:teuthology.orchestra.run.smithi102.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-08-19T14:07:37.105 INFO:teuthology.orchestra.run.smithi102.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-08-19T14:07:37.121 INFO:teuthology.orchestra.run.smithi102.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 394 kB/s | 16 kB 00:00 2024-08-19T14:07:37.147 INFO:teuthology.orchestra.run.smithi102.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-08-19T14:07:37.172 INFO:teuthology.orchestra.run.smithi102.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-08-19T14:07:37.214 INFO:teuthology.orchestra.run.smithi102.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-08-19T14:07:37.240 INFO:teuthology.orchestra.run.smithi102.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 798 kB/s | 20 kB 00:00 2024-08-19T14:07:37.265 INFO:teuthology.orchestra.run.smithi102.stdout:(36/37): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-08-19T14:07:37.307 INFO:teuthology.orchestra.run.smithi102.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-08-19T14:07:37.311 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:07:37.311 INFO:teuthology.orchestra.run.smithi102.stdout:Total 8.6 MB/s | 30 MB 00:03 2024-08-19T14:07:37.463 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-08-19T14:07:37.503 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-08-19T14:07:37.503 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-08-19T14:07:37.800 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-08-19T14:07:37.801 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-08-19T14:07:39.168 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-08-19T14:07:39.227 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-08-19T14:07:39.259 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-08-19T14:07:39.287 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-08-19T14:07:39.358 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-08-19T14:07:39.411 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-08-19T14:07:39.444 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-08-19T14:07:39.537 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-08-19T14:07:39.622 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-08-19T14:07:39.663 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/37 2024-08-19T14:07:39.698 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-08-19T14:07:39.728 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-08-19T14:07:39.763 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-08-19T14:07:39.796 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-08-19T14:07:39.837 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-08-19T14:07:39.872 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-08-19T14:07:39.906 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-08-19T14:07:39.954 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-08-19T14:07:40.003 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-08-19T14:07:40.077 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-08-19T14:07:40.215 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-08-19T14:07:40.295 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-08-19T14:07:40.357 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-08-19T14:07:40.395 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-08-19T14:07:40.453 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-08-19T14:07:40.508 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-08-19T14:07:40.694 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-08-19T14:07:40.741 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-08-19T14:07:40.819 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-08-19T14:07:40.908 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-08-19T14:07:40.949 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-08-19T14:07:41.065 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-08-19T14:07:41.221 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-modules-core-2:19.1.0-1382.gde09dd01.el9. 32/37 2024-08-19T14:07:41.251 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-2:19.1.0-1382.gde09dd01.el9.x86_64 33/37 2024-08-19T14:07:41.278 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-2:19.1.0-1382.gde09dd01.el9.x86_64 33/37 2024-08-19T14:07:41.278 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:07:41.278 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-19T14:07:41.278 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T14:07:41.278 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T14:07:41.278 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:42.727 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-osd-2:19.1.0-1382.gde09dd01.el9.x86_64 34/37 2024-08-19T14:07:42.754 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-osd-2:19.1.0-1382.gde09dd01.el9.x86_64 34/37 2024-08-19T14:07:42.754 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:07:42.755 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-19T14:07:42.755 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T14:07:42.755 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T14:07:42.755 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:43.186 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mon-2:19.1.0-1382.gde09dd01.el9.x86_64 35/37 2024-08-19T14:07:43.209 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mon-2:19.1.0-1382.gde09dd01.el9.x86_64 35/37 2024-08-19T14:07:43.209 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:07:43.209 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-19T14:07:43.210 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T14:07:43.210 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T14:07:43.210 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:43.436 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mds-2:19.1.0-1382.gde09dd01.el9.x86_64 36/37 2024-08-19T14:07:43.460 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mds-2:19.1.0-1382.gde09dd01.el9.x86_64 36/37 2024-08-19T14:07:43.460 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:07:43.460 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-19T14:07:43.460 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T14:07:43.460 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T14:07:43.460 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:43.524 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-2:19.1.0-1382.gde09dd01.el9.x86_64 37/37 2024-08-19T14:07:45.172 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-2:19.1.0-1382.gde09dd01.el9.x86_64 37/37 2024-08-19T14:07:45.172 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-2:19.1.0-1382.gde09dd01.el9.x86_64 1/37 2024-08-19T14:07:45.172 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mds-2:19.1.0-1382.gde09dd01.el9.x86_64 2/37 2024-08-19T14:07:45.172 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-2:19.1.0-1382.gde09dd01.el9.x86_64 3/37 2024-08-19T14:07:45.172 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mon-2:19.1.0-1382.gde09dd01.el9.x86_64 4/37 2024-08-19T14:07:45.172 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-osd-2:19.1.0-1382.gde09dd01.el9.x86_64 5/37 2024-08-19T14:07:45.173 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-modules-core-2:19.1.0-1382.gde09dd01.el9. 6/37 2024-08-19T14:07:45.173 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-08-19T14:07:45.173 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-08-19T14:07:45.173 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-08-19T14:07:45.173 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-08-19T14:07:45.173 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-08-19T14:07:45.173 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-08-19T14:07:45.173 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-08-19T14:07:45.173 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-08-19T14:07:45.173 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-08-19T14:07:45.173 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-08-19T14:07:45.174 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-08-19T14:07:45.174 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-08-19T14:07:45.174 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-08-19T14:07:45.174 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-08-19T14:07:45.174 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-08-19T14:07:45.174 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-08-19T14:07:45.174 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-08-19T14:07:45.174 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-08-19T14:07:45.174 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-08-19T14:07:45.174 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-08-19T14:07:45.174 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-08-19T14:07:45.174 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-08-19T14:07:45.174 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-08-19T14:07:45.175 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-08-19T14:07:45.175 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-08-19T14:07:45.175 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-08-19T14:07:45.175 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-08-19T14:07:45.175 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-08-19T14:07:45.175 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 35/37 2024-08-19T14:07:45.175 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-08-19T14:07:46.182 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-08-19T14:07:46.182 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:46.182 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-08-19T14:07:46.182 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:46.182 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:46.183 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:46.183 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T14:07:46.183 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:46.183 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:46.183 INFO:teuthology.orchestra.run.smithi102.stdout: lua-5.4.4-4.el9.x86_64 2024-08-19T14:07:46.183 INFO:teuthology.orchestra.run.smithi102.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-19T14:07:46.183 INFO:teuthology.orchestra.run.smithi102.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-19T14:07:46.183 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-19T14:07:46.183 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-19T14:07:46.183 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-19T14:07:46.183 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-19T14:07:46.183 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-19T14:07:46.184 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-19T14:07:46.184 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-19T14:07:46.184 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-19T14:07:46.184 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-19T14:07:46.184 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-19T14:07:46.184 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-19T14:07:46.184 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-19T14:07:46.184 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-19T14:07:46.184 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-19T14:07:46.184 INFO:teuthology.orchestra.run.smithi102.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-19T14:07:46.184 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-19T14:07:46.184 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply-3.11-14.el9.noarch 2024-08-19T14:07:46.184 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-19T14:07:46.185 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-19T14:07:46.185 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-19T14:07:46.185 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-19T14:07:46.185 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-19T14:07:46.185 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-19T14:07:46.185 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-19T14:07:46.185 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-19T14:07:46.185 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-19T14:07:46.185 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-19T14:07:46.185 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-19T14:07:46.185 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:46.185 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:07:46.545 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-base 2024-08-19T14:07:47.089 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:52 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:07:47.173 INFO:teuthology.orchestra.run.smithi102.stdout:Package ceph-base-2:19.1.0-1382.gde09dd01.el9.x86_64 is already installed. 2024-08-19T14:07:47.215 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:07:47.216 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-08-19T14:07:47.216 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:07:47.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install cephadm 2024-08-19T14:07:47.809 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:52 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:07:47.936 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:07:47.936 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:07:47.937 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-08-19T14:07:47.937 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:07:47.937 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-08-19T14:07:47.937 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 766 k 2024-08-19T14:07:47.937 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:47.937 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-08-19T14:07:47.937 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:07:47.937 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-08-19T14:07:47.937 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:47.937 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 766 k 2024-08-19T14:07:47.937 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 772 k 2024-08-19T14:07:47.938 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-08-19T14:07:48.219 INFO:teuthology.orchestra.run.smithi102.stdout:cephadm-19.1.0-1382.gde09dd01.el9.noarch.rpm 2.7 MB/s | 766 kB 00:00 2024-08-19T14:07:48.220 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:07:48.220 INFO:teuthology.orchestra.run.smithi102.stdout:Total 2.6 MB/s | 766 kB 00:00 2024-08-19T14:07:48.221 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-08-19T14:07:48.226 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-08-19T14:07:48.226 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-08-19T14:07:48.232 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-08-19T14:07:48.232 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-08-19T14:07:48.307 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-08-19T14:07:48.730 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 1/1 2024-08-19T14:07:48.823 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 1/1 2024-08-19T14:07:49.177 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 1/1 2024-08-19T14:07:49.436 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 1/1 2024-08-19T14:07:49.437 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:49.437 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-08-19T14:07:49.437 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T14:07:49.437 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:49.437 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:07:49.655 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-immutable-object-cache 2024-08-19T14:07:50.196 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:55 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:07:50.324 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:07:50.324 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:07:50.325 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-08-19T14:07:50.325 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:07:50.325 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-08-19T14:07:50.325 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 145 k 2024-08-19T14:07:50.325 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:50.325 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-08-19T14:07:50.325 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:07:50.325 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-08-19T14:07:50.325 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:50.325 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 145 k 2024-08-19T14:07:50.325 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 435 k 2024-08-19T14:07:50.326 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-08-19T14:07:50.535 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-immutable-object-cache-19.1.0-1382.gde09dd 694 kB/s | 145 kB 00:00 2024-08-19T14:07:50.535 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:07:50.536 INFO:teuthology.orchestra.run.smithi102.stdout:Total 688 kB/s | 145 kB 00:00 2024-08-19T14:07:50.536 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-08-19T14:07:50.543 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-08-19T14:07:50.543 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-08-19T14:07:50.599 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-08-19T14:07:50.599 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-08-19T14:07:50.750 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-08-19T14:07:50.827 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-immutable-object-cache-2:19.1.0-1382.gde09dd01. 1/1 2024-08-19T14:07:50.852 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.0-1382.gde09dd01. 1/1 2024-08-19T14:07:50.853 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:07:50.853 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-19T14:07:50.853 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:51.762 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-immutable-object-cache-2:19.1.0-1382.gde09dd01. 1/1 2024-08-19T14:07:51.762 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:51.762 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-08-19T14:07:51.762 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:07:51.762 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:51.762 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:07:51.972 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr 2024-08-19T14:07:52.517 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:57 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:07:52.602 INFO:teuthology.orchestra.run.smithi102.stdout:Package ceph-mgr-2:19.1.0-1382.gde09dd01.el9.x86_64 is already installed. 2024-08-19T14:07:52.646 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:07:52.647 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-08-19T14:07:52.647 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:07:52.710 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-dashboard 2024-08-19T14:07:53.242 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:58 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:07:53.367 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:07:53.368 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:07:53.369 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-08-19T14:07:53.369 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:07:53.369 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-08-19T14:07:53.369 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 3.6 M 2024-08-19T14:07:53.369 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-08-19T14:07:53.369 INFO:teuthology.orchestra.run.smithi102.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-19T14:07:53.369 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 32 k 2024-08-19T14:07:53.369 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 16 k 2024-08-19T14:07:53.369 INFO:teuthology.orchestra.run.smithi102.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-19T14:07:53.369 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-19T14:07:53.369 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-19T14:07:53.369 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-19T14:07:53.369 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-19T14:07:53.370 INFO:teuthology.orchestra.run.smithi102.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-19T14:07:53.370 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-19T14:07:53.370 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-19T14:07:53.370 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:53.370 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-08-19T14:07:53.370 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:07:53.370 INFO:teuthology.orchestra.run.smithi102.stdout:Install 12 Packages 2024-08-19T14:07:53.370 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:53.371 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 8.7 M 2024-08-19T14:07:53.371 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 104 M 2024-08-19T14:07:53.371 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-08-19T14:07:53.797 INFO:teuthology.orchestra.run.smithi102.stdout:(1/12): ceph-grafana-dashboards-19.1.0-1382.gde 222 kB/s | 32 kB 00:00 2024-08-19T14:07:53.815 INFO:teuthology.orchestra.run.smithi102.stdout:(2/12): ceph-prometheus-alerts-19.1.0-1382.gde0 103 kB/s | 16 kB 00:00 2024-08-19T14:07:54.023 INFO:teuthology.orchestra.run.smithi102.stdout:(3/12): ceph-mgr-dashboard-19.1.0-1382.gde09dd0 9.6 MB/s | 3.6 MB 00:00 2024-08-19T14:07:54.207 INFO:teuthology.orchestra.run.smithi102.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.6 MB/s | 863 kB 00:00 2024-08-19T14:07:54.265 INFO:teuthology.orchestra.run.smithi102.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.4 MB/s | 548 kB 00:00 2024-08-19T14:07:54.282 INFO:teuthology.orchestra.run.smithi102.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-08-19T14:07:54.399 INFO:teuthology.orchestra.run.smithi102.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 18 MB/s | 2.0 MB 00:00 2024-08-19T14:07:54.425 INFO:teuthology.orchestra.run.smithi102.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-08-19T14:07:54.450 INFO:teuthology.orchestra.run.smithi102.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-08-19T14:07:54.475 INFO:teuthology.orchestra.run.smithi102.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.3 MB/s | 188 kB 00:00 2024-08-19T14:07:54.684 INFO:teuthology.orchestra.run.smithi102.stdout:(11/12): python3-protobuf-3.14.0-13.el9.noarch. 310 kB/s | 269 kB 00:00 2024-08-19T14:07:54.834 INFO:teuthology.orchestra.run.smithi102.stdout:(12/12): protobuf-3.14.0-13.el9.x86_64.rpm 997 kB/s | 1.0 MB 00:01 2024-08-19T14:07:54.837 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:07:54.838 INFO:teuthology.orchestra.run.smithi102.stdout:Total 5.9 MB/s | 8.7 MB 00:01 2024-08-19T14:07:54.886 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-08-19T14:07:54.902 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-08-19T14:07:54.902 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-08-19T14:07:55.057 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-08-19T14:07:55.058 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-08-19T14:07:55.610 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-08-19T14:07:55.693 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-08-19T14:07:55.736 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-08-19T14:07:55.782 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-08-19T14:07:55.813 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-08-19T14:07:55.888 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-08-19T14:07:56.021 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-08-19T14:07:56.124 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-08-19T14:07:56.168 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-08-19T14:07:56.197 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-08-19T14:07:56.228 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-prometheus-alerts-2:19.1.0-1382.gde09dd01.el9 10/12 2024-08-19T14:07:57.278 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-grafana-dashboards-2:19.1.0-1382.gde09dd01.el 11/12 2024-08-19T14:07:57.330 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-dashboard-2:19.1.0-1382.gde09dd01.el9.noa 12/12 2024-08-19T14:07:58.601 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.0-1382.gde09dd01.el9.noa 12/12 2024-08-19T14:07:58.601 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-grafana-dashboards-2:19.1.0-1382.gde09dd01.el 1/12 2024-08-19T14:07:58.601 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-dashboard-2:19.1.0-1382.gde09dd01.el9.noa 2/12 2024-08-19T14:07:58.601 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-prometheus-alerts-2:19.1.0-1382.gde09dd01.el9 3/12 2024-08-19T14:07:58.601 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-08-19T14:07:58.601 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-08-19T14:07:58.601 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-08-19T14:07:58.601 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-08-19T14:07:58.601 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-08-19T14:07:58.602 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-08-19T14:07:58.602 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-08-19T14:07:58.602 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-08-19T14:07:58.902 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-08-19T14:07:58.902 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:58.902 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-08-19T14:07:58.902 INFO:teuthology.orchestra.run.smithi102.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-19T14:07:58.902 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T14:07:58.903 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T14:07:58.903 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T14:07:58.903 INFO:teuthology.orchestra.run.smithi102.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-19T14:07:58.903 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-19T14:07:58.903 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-19T14:07:58.903 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-19T14:07:58.903 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-19T14:07:58.903 INFO:teuthology.orchestra.run.smithi102.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-19T14:07:58.903 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-19T14:07:58.903 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-19T14:07:58.903 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:58.904 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:07:59.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-19T14:07:59.689 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:04 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:07:59.816 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:07:59.817 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================= 2024-08-19T14:07:59.817 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-08-19T14:07:59.817 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================= 2024-08-19T14:07:59.817 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-08-19T14:07:59.817 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 7.4 M 2024-08-19T14:07:59.817 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-08-19T14:07:59.817 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-19T14:07:59.817 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-19T14:07:59.817 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-19T14:07:59.818 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-19T14:07:59.818 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-19T14:07:59.818 INFO:teuthology.orchestra.run.smithi102.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-19T14:07:59.818 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-19T14:07:59.818 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel x86_64 3.9.19-4.el9 appstream 244 k 2024-08-19T14:07:59.818 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-19T14:07:59.818 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-19T14:07:59.818 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-19T14:07:59.818 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:59.818 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-08-19T14:07:59.819 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================= 2024-08-19T14:07:59.819 INFO:teuthology.orchestra.run.smithi102.stdout:Install 12 Packages 2024-08-19T14:07:59.819 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:07:59.819 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 38 M 2024-08-19T14:07:59.820 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 209 M 2024-08-19T14:07:59.820 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-08-19T14:08:00.634 INFO:teuthology.orchestra.run.smithi102.stdout:(1/12): ceph-mgr-diskprediction-local-19.1.0-13 16 MB/s | 7.4 MB 00:00 2024-08-19T14:08:00.760 INFO:teuthology.orchestra.run.smithi102.stdout:(2/12): libquadmath-11.4.1-3.el9.x86_64.rpm 334 kB/s | 195 kB 00:00 2024-08-19T14:08:00.785 INFO:teuthology.orchestra.run.smithi102.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 218 kB/s | 33 kB 00:00 2024-08-19T14:08:00.843 INFO:teuthology.orchestra.run.smithi102.stdout:(4/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-08-19T14:08:00.894 INFO:teuthology.orchestra.run.smithi102.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 167 kB/s | 18 kB 00:00 2024-08-19T14:08:00.961 INFO:teuthology.orchestra.run.smithi102.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 15 MB/s | 3.0 MB 00:00 2024-08-19T14:08:00.986 INFO:teuthology.orchestra.run.smithi102.stdout:(7/12): openblas-0.3.21-2.el9.x86_64.rpm 247 kB/s | 35 kB 00:00 2024-08-19T14:08:01.045 INFO:teuthology.orchestra.run.smithi102.stdout:(8/12): python3-devel-3.9.19-4.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-08-19T14:08:01.120 INFO:teuthology.orchestra.run.smithi102.stdout:(9/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-08-19T14:08:01.145 INFO:teuthology.orchestra.run.smithi102.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.4 MB/s | 250 kB 00:00 2024-08-19T14:08:01.287 INFO:teuthology.orchestra.run.smithi102.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-08-19T14:08:01.721 INFO:teuthology.orchestra.run.smithi102.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-08-19T14:08:01.721 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:08:01.721 INFO:teuthology.orchestra.run.smithi102.stdout:Total 20 MB/s | 38 MB 00:01 2024-08-19T14:08:02.078 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-08-19T14:08:02.104 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-08-19T14:08:02.104 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-08-19T14:08:02.514 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-08-19T14:08:02.515 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-08-19T14:08:03.297 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-08-19T14:08:03.359 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-19T14:08:03.402 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-19T14:08:03.504 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-19T14:08:03.545 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-devel-3.9.19-4.el9.x86_64 4/12 2024-08-19T14:08:03.722 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-19T14:08:03.752 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-19T14:08:03.828 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-19T14:08:04.468 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-19T14:08:04.582 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-19T14:08:05.793 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-19T14:08:06.830 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-19T14:08:06.861 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.0-1382.gde09d 12/12 2024-08-19T14:08:09.296 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.0-1382.gde09d 12/12 2024-08-19T14:08:09.296 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.0-1382.gde09d 1/12 2024-08-19T14:08:09.296 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-19T14:08:09.296 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-19T14:08:09.296 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-19T14:08:09.296 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-19T14:08:09.296 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-19T14:08:09.296 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-19T14:08:09.296 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-19T14:08:09.297 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-devel-3.9.19-4.el9.x86_64 9/12 2024-08-19T14:08:09.297 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-19T14:08:09.297 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-19T14:08:09.687 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-19T14:08:09.687 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:09.687 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-08-19T14:08:09.687 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T14:08:09.687 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-19T14:08:09.687 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-19T14:08:09.687 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-19T14:08:09.687 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-19T14:08:09.687 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-19T14:08:09.687 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-19T14:08:09.688 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-19T14:08:09.688 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel-3.9.19-4.el9.x86_64 2024-08-19T14:08:09.688 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-19T14:08:09.688 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-19T14:08:09.688 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-19T14:08:09.688 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:09.688 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:08:10.068 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-rook 2024-08-19T14:08:10.617 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:15 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:08:10.744 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:08:10.745 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:10.745 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-08-19T14:08:10.745 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:10.746 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-08-19T14:08:10.746 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 49 k 2024-08-19T14:08:10.746 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-08-19T14:08:10.746 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-19T14:08:10.746 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-19T14:08:10.746 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-19T14:08:10.746 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-19T14:08:10.746 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-19T14:08:10.746 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-19T14:08:10.746 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-19T14:08:10.747 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-19T14:08:10.747 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-19T14:08:10.747 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-19T14:08:10.747 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-19T14:08:10.747 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-19T14:08:10.747 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:10.747 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-08-19T14:08:10.747 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:10.747 INFO:teuthology.orchestra.run.smithi102.stdout:Install 13 Packages 2024-08-19T14:08:10.747 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:10.748 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 2.2 M 2024-08-19T14:08:10.748 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 26 M 2024-08-19T14:08:10.748 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-08-19T14:08:11.062 INFO:teuthology.orchestra.run.smithi102.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 230 kB/s | 19 kB 00:00 2024-08-19T14:08:11.078 INFO:teuthology.orchestra.run.smithi102.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 256 kB/s | 26 kB 00:00 2024-08-19T14:08:11.137 INFO:teuthology.orchestra.run.smithi102.stdout:(3/13): ceph-mgr-rook-19.1.0-1382.gde09dd01.el9 310 kB/s | 49 kB 00:00 2024-08-19T14:08:11.187 INFO:teuthology.orchestra.run.smithi102.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-08-19T14:08:11.212 INFO:teuthology.orchestra.run.smithi102.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-08-19T14:08:11.238 INFO:teuthology.orchestra.run.smithi102.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-08-19T14:08:11.263 INFO:teuthology.orchestra.run.smithi102.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 639 kB/s | 32 kB 00:00 2024-08-19T14:08:11.280 INFO:teuthology.orchestra.run.smithi102.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 351 kB/s | 14 kB 00:00 2024-08-19T14:08:11.305 INFO:teuthology.orchestra.run.smithi102.stdout:(9/13): python3-google-auth-2.33.0-1.el9.noarch 5.3 MB/s | 220 kB 00:00 2024-08-19T14:08:11.331 INFO:teuthology.orchestra.run.smithi102.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-08-19T14:08:11.356 INFO:teuthology.orchestra.run.smithi102.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-08-19T14:08:11.381 INFO:teuthology.orchestra.run.smithi102.stdout:(12/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.1 MB/s | 279 kB 00:00 2024-08-19T14:08:11.423 INFO:teuthology.orchestra.run.smithi102.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.1 MB/s | 1.0 MB 00:00 2024-08-19T14:08:11.425 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:08:11.425 INFO:teuthology.orchestra.run.smithi102.stdout:Total 3.3 MB/s | 2.2 MB 00:00 2024-08-19T14:08:11.491 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-08-19T14:08:11.505 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-08-19T14:08:11.506 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-08-19T14:08:11.591 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-08-19T14:08:11.592 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-08-19T14:08:12.100 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-08-19T14:08:12.261 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-19T14:08:12.316 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-19T14:08:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-19T14:08:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-19T14:08:12.458 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-19T14:08:12.559 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-19T14:08:12.670 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 7/13 2024-08-19T14:08:12.718 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-19T14:08:13.424 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-19T14:08:13.483 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-19T14:08:13.517 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-19T14:08:13.564 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-19T14:08:13.587 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-rook-2:19.1.0-1382.gde09dd01.el9.noarch 13/13 2024-08-19T14:08:14.426 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.0-1382.gde09dd01.el9.noarch 13/13 2024-08-19T14:08:14.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-rook-2:19.1.0-1382.gde09dd01.el9.noarch 1/13 2024-08-19T14:08:14.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-19T14:08:14.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-19T14:08:14.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-19T14:08:14.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-19T14:08:14.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-19T14:08:14.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-19T14:08:14.426 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-19T14:08:14.427 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-19T14:08:14.427 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 10/13 2024-08-19T14:08:14.427 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-19T14:08:14.427 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-19T14:08:14.741 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-19T14:08:14.741 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:14.741 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-08-19T14:08:14.741 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T14:08:14.741 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-19T14:08:14.741 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-19T14:08:14.741 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-19T14:08:14.741 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-19T14:08:14.741 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-19T14:08:14.742 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-19T14:08:14.742 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-19T14:08:14.742 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-19T14:08:14.742 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-19T14:08:14.742 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-19T14:08:14.742 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-19T14:08:14.742 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-19T14:08:14.742 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:14.742 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:08:14.965 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-cephadm 2024-08-19T14:08:15.500 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:20 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:08:15.627 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:08:15.627 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:15.627 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-08-19T14:08:15.628 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:15.628 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-08-19T14:08:15.628 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 146 k 2024-08-19T14:08:15.628 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-08-19T14:08:15.628 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-19T14:08:15.628 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-19T14:08:15.628 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-19T14:08:15.628 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-19T14:08:15.628 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-19T14:08:15.628 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:15.628 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-08-19T14:08:15.628 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:15.628 INFO:teuthology.orchestra.run.smithi102.stdout:Install 6 Packages 2024-08-19T14:08:15.629 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:15.629 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 7.0 M 2024-08-19T14:08:15.629 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 33 M 2024-08-19T14:08:15.629 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-08-19T14:08:16.110 INFO:teuthology.orchestra.run.smithi102.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-08-19T14:08:16.135 INFO:teuthology.orchestra.run.smithi102.stdout:(2/6): ceph-mgr-cephadm-19.1.0-1382.gde09dd01.e 771 kB/s | 146 kB 00:00 2024-08-19T14:08:16.186 INFO:teuthology.orchestra.run.smithi102.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.3 MB/s | 58 kB 00:00 2024-08-19T14:08:16.219 INFO:teuthology.orchestra.run.smithi102.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-08-19T14:08:16.244 INFO:teuthology.orchestra.run.smithi102.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.0 MB/s | 548 kB 00:00 2024-08-19T14:08:16.320 INFO:teuthology.orchestra.run.smithi102.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-08-19T14:08:16.322 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:08:16.323 INFO:teuthology.orchestra.run.smithi102.stdout:Total 10 MB/s | 7.0 MB 00:00 2024-08-19T14:08:16.393 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-08-19T14:08:16.404 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-08-19T14:08:16.404 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-08-19T14:08:16.501 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-08-19T14:08:16.501 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-08-19T14:08:16.805 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-08-19T14:08:16.968 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-19T14:08:17.021 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-19T14:08:17.544 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-19T14:08:17.626 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-19T14:08:17.690 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-19T14:08:17.719 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 6/6 2024-08-19T14:08:18.488 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 6/6 2024-08-19T14:08:18.488 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 1/6 2024-08-19T14:08:18.489 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-19T14:08:18.489 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-19T14:08:18.489 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-19T14:08:18.489 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-19T14:08:18.749 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-19T14:08:18.749 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:18.749 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-08-19T14:08:18.749 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T14:08:18.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-19T14:08:18.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-19T14:08:18.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-19T14:08:18.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-19T14:08:18.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-19T14:08:18.749 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:18.750 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:08:19.001 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-fuse 2024-08-19T14:08:19.540 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:24 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:08:19.667 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:08:19.668 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:19.668 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-08-19T14:08:19.668 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:19.668 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-08-19T14:08:19.668 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 896 k 2024-08-19T14:08:19.668 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-08-19T14:08:19.668 INFO:teuthology.orchestra.run.smithi102.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-19T14:08:19.668 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:19.668 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-08-19T14:08:19.668 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:19.669 INFO:teuthology.orchestra.run.smithi102.stdout:Install 2 Packages 2024-08-19T14:08:19.669 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:19.669 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 976 k 2024-08-19T14:08:19.669 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 2.9 M 2024-08-19T14:08:19.669 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-08-19T14:08:19.915 INFO:teuthology.orchestra.run.smithi102.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 604 kB/s | 80 kB 00:00 2024-08-19T14:08:20.048 INFO:teuthology.orchestra.run.smithi102.stdout:(2/2): ceph-fuse-19.1.0-1382.gde09dd01.el9.x86_ 3.3 MB/s | 896 kB 00:00 2024-08-19T14:08:20.049 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:08:20.049 INFO:teuthology.orchestra.run.smithi102.stdout:Total 2.5 MB/s | 976 kB 00:00 2024-08-19T14:08:20.061 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-08-19T14:08:20.082 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-08-19T14:08:20.082 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-08-19T14:08:20.161 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-08-19T14:08:20.162 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-08-19T14:08:20.340 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-08-19T14:08:20.477 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-19T14:08:20.539 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 2/2 2024-08-19T14:08:21.179 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 2/2 2024-08-19T14:08:21.179 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 1/2 2024-08-19T14:08:21.442 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-19T14:08:21.442 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:21.442 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-08-19T14:08:21.442 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-19T14:08:21.442 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:21.442 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:08:21.719 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-volume 2024-08-19T14:08:22.264 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:27 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:08:22.393 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:08:22.394 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:22.394 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-08-19T14:08:22.394 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:22.394 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-08-19T14:08:22.394 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 285 k 2024-08-19T14:08:22.394 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:22.394 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-08-19T14:08:22.394 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:22.394 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-08-19T14:08:22.395 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:22.395 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 285 k 2024-08-19T14:08:22.395 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 1.3 M 2024-08-19T14:08:22.395 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-08-19T14:08:22.604 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-volume-19.1.0-1382.gde09dd01.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-08-19T14:08:22.605 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:08:22.605 INFO:teuthology.orchestra.run.smithi102.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-08-19T14:08:22.605 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-08-19T14:08:22.613 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-08-19T14:08:22.613 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-08-19T14:08:22.635 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-08-19T14:08:22.635 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-08-19T14:08:22.851 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-08-19T14:08:22.914 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-volume-2:19.1.0-1382.gde09dd01.el9.noarch 1/1 2024-08-19T14:08:22.931 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-volume-2:19.1.0-1382.gde09dd01.el9.noarch 1/1 2024-08-19T14:08:22.931 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:08:22.931 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-19T14:08:22.931 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:23.965 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-volume-2:19.1.0-1382.gde09dd01.el9.noarch 1/1 2024-08-19T14:08:23.966 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:23.966 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-08-19T14:08:23.966 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T14:08:23.966 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:23.966 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:08:24.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librados-devel 2024-08-19T14:08:24.783 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:29 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:08:24.916 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:08:24.917 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:24.917 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-08-19T14:08:24.917 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:24.917 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-08-19T14:08:24.917 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 127 k 2024-08-19T14:08:24.917 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:24.917 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-08-19T14:08:24.917 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:24.917 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-08-19T14:08:24.918 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:24.918 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 127 k 2024-08-19T14:08:24.918 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 456 k 2024-08-19T14:08:24.918 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-08-19T14:08:25.118 INFO:teuthology.orchestra.run.smithi102.stdout:librados-devel-19.1.0-1382.gde09dd01.el9.x86_64 633 kB/s | 127 kB 00:00 2024-08-19T14:08:25.118 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:08:25.118 INFO:teuthology.orchestra.run.smithi102.stdout:Total 629 kB/s | 127 kB 00:00 2024-08-19T14:08:25.119 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-08-19T14:08:25.124 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-08-19T14:08:25.124 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-08-19T14:08:25.188 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-08-19T14:08:25.188 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-08-19T14:08:25.332 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-08-19T14:08:25.430 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librados-devel-2:19.1.0-1382.gde09dd01.el9.x86_64 1/1 2024-08-19T14:08:25.865 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados-devel-2:19.1.0-1382.gde09dd01.el9.x86_64 1/1 2024-08-19T14:08:26.122 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados-devel-2:19.1.0-1382.gde09dd01.el9.x86_64 1/1 2024-08-19T14:08:26.122 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:26.122 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-08-19T14:08:26.122 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:08:26.122 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:26.122 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:08:26.369 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install libcephfs2 2024-08-19T14:08:26.916 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:31 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:08:27.001 INFO:teuthology.orchestra.run.smithi102.stdout:Package libcephfs2-2:19.1.0-1382.gde09dd01.el9.x86_64 is already installed. 2024-08-19T14:08:27.047 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:08:27.048 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-08-19T14:08:27.048 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:08:27.109 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install libcephfs-devel 2024-08-19T14:08:27.648 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:32 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:08:27.775 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:08:27.776 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:27.776 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-08-19T14:08:27.776 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:27.776 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-08-19T14:08:27.776 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 32 k 2024-08-19T14:08:27.777 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:27.777 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-08-19T14:08:27.777 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:27.777 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-08-19T14:08:27.777 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:27.777 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 32 k 2024-08-19T14:08:27.777 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 142 k 2024-08-19T14:08:27.777 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-08-19T14:08:27.929 INFO:teuthology.orchestra.run.smithi102.stdout:libcephfs-devel-19.1.0-1382.gde09dd01.el9.x86_6 208 kB/s | 32 kB 00:00 2024-08-19T14:08:27.930 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:08:27.930 INFO:teuthology.orchestra.run.smithi102.stdout:Total 206 kB/s | 32 kB 00:00 2024-08-19T14:08:27.930 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-08-19T14:08:27.934 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-08-19T14:08:27.934 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-08-19T14:08:27.937 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-08-19T14:08:27.937 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-08-19T14:08:28.022 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-08-19T14:08:28.118 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs-devel-2:19.1.0-1382.gde09dd01.el9.x86_64 1/1 2024-08-19T14:08:28.428 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs-devel-2:19.1.0-1382.gde09dd01.el9.x86_64 1/1 2024-08-19T14:08:28.638 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs-devel-2:19.1.0-1382.gde09dd01.el9.x86_64 1/1 2024-08-19T14:08:28.638 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:28.638 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-08-19T14:08:28.638 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:08:28.638 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:28.638 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:08:28.813 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librados2 2024-08-19T14:08:29.350 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:34 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:08:29.433 INFO:teuthology.orchestra.run.smithi102.stdout:Package librados2-2:19.1.0-1382.gde09dd01.el9.x86_64 is already installed. 2024-08-19T14:08:29.476 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:08:29.477 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-08-19T14:08:29.477 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:08:29.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librbd1 2024-08-19T14:08:30.074 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:35 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:08:30.158 INFO:teuthology.orchestra.run.smithi102.stdout:Package librbd1-2:19.1.0-1382.gde09dd01.el9.x86_64 is already installed. 2024-08-19T14:08:30.202 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:08:30.203 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-08-19T14:08:30.203 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:08:30.277 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rados 2024-08-19T14:08:30.815 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:35 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:08:30.898 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rados-2:19.1.0-1382.gde09dd01.el9.x86_64 is already installed. 2024-08-19T14:08:30.940 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:08:30.941 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-08-19T14:08:30.941 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:08:31.001 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rgw 2024-08-19T14:08:31.538 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:36 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:08:31.623 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rgw-2:19.1.0-1382.gde09dd01.el9.x86_64 is already installed. 2024-08-19T14:08:31.666 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:08:31.667 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-08-19T14:08:31.667 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:08:31.725 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-cephfs 2024-08-19T14:08:32.255 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:37 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:08:32.340 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-cephfs-2:19.1.0-1382.gde09dd01.el9.x86_64 is already installed. 2024-08-19T14:08:32.383 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:08:32.384 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-08-19T14:08:32.384 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:08:32.442 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rbd 2024-08-19T14:08:32.978 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:37 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:08:33.061 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rbd-2:19.1.0-1382.gde09dd01.el9.x86_64 is already installed. 2024-08-19T14:08:33.104 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:08:33.105 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-08-19T14:08:33.105 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:08:33.170 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-fuse 2024-08-19T14:08:33.705 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:38 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:08:33.834 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:08:33.834 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:33.835 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-08-19T14:08:33.835 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:33.835 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-08-19T14:08:33.835 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 85 k 2024-08-19T14:08:33.835 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:33.835 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-08-19T14:08:33.835 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:33.835 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-08-19T14:08:33.835 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:33.835 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 85 k 2024-08-19T14:08:33.835 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 231 k 2024-08-19T14:08:33.835 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-08-19T14:08:34.025 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-fuse-19.1.0-1382.gde09dd01.el9.x86_64.rpm 449 kB/s | 85 kB 00:00 2024-08-19T14:08:34.026 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:08:34.026 INFO:teuthology.orchestra.run.smithi102.stdout:Total 445 kB/s | 85 kB 00:00 2024-08-19T14:08:34.026 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-08-19T14:08:34.033 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-08-19T14:08:34.033 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-08-19T14:08:34.096 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-08-19T14:08:34.096 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-08-19T14:08:34.235 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-08-19T14:08:34.346 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 1/1 2024-08-19T14:08:34.698 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 1/1 2024-08-19T14:08:34.939 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 1/1 2024-08-19T14:08:34.940 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:34.940 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-08-19T14:08:34.940 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:08:34.940 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:34.940 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:08:35.208 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-mirror 2024-08-19T14:08:35.755 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:40 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:08:35.885 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:08:35.885 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:35.885 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-08-19T14:08:35.886 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:35.886 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-08-19T14:08:35.886 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 3.1 M 2024-08-19T14:08:35.886 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:35.886 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-08-19T14:08:35.886 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:35.886 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-08-19T14:08:35.886 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:35.886 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 3.1 M 2024-08-19T14:08:35.886 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 13 M 2024-08-19T14:08:35.886 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-08-19T14:08:36.228 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-mirror-19.1.0-1382.gde09dd01.el9.x86_64.rpm 9.1 MB/s | 3.1 MB 00:00 2024-08-19T14:08:36.229 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:08:36.229 INFO:teuthology.orchestra.run.smithi102.stdout:Total 9.0 MB/s | 3.1 MB 00:00 2024-08-19T14:08:36.230 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-08-19T14:08:36.239 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-08-19T14:08:36.239 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-08-19T14:08:36.319 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-08-19T14:08:36.320 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-08-19T14:08:36.742 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-08-19T14:08:36.804 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 1/1 2024-08-19T14:08:36.829 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 1/1 2024-08-19T14:08:36.829 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:08:36.829 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-19T14:08:36.829 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T14:08:36.829 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T14:08:36.829 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:37.880 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 1/1 2024-08-19T14:08:37.880 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:37.880 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-08-19T14:08:37.881 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:08:37.881 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:37.881 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:08:38.099 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-nbd 2024-08-19T14:08:38.643 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:43 ago on Mon 19 Aug 2024 02:06:55 PM UTC. 2024-08-19T14:08:38.772 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-08-19T14:08:38.773 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:38.773 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-08-19T14:08:38.773 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:38.773 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-08-19T14:08:38.773 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 170 k 2024-08-19T14:08:38.773 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:38.773 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-08-19T14:08:38.773 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-08-19T14:08:38.773 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-08-19T14:08:38.774 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:38.774 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 170 k 2024-08-19T14:08:38.774 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 494 k 2024-08-19T14:08:38.774 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-08-19T14:08:38.974 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-nbd-19.1.0-1382.gde09dd01.el9.x86_64.rpm 851 kB/s | 170 kB 00:00 2024-08-19T14:08:38.975 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:08:38.975 INFO:teuthology.orchestra.run.smithi102.stdout:Total 846 kB/s | 170 kB 00:00 2024-08-19T14:08:38.975 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-08-19T14:08:38.982 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-08-19T14:08:38.983 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-08-19T14:08:39.046 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-08-19T14:08:39.046 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-08-19T14:08:39.201 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-08-19T14:08:39.307 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-nbd-2:19.1.0-1382.gde09dd01.el9.x86_64 1/1 2024-08-19T14:08:39.728 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-nbd-2:19.1.0-1382.gde09dd01.el9.x86_64 1/1 2024-08-19T14:08:40.002 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-nbd-2:19.1.0-1382.gde09dd01.el9.x86_64 1/1 2024-08-19T14:08:40.002 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:40.002 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-08-19T14:08:40.002 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T14:08:40.002 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-08-19T14:08:40.002 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-08-19T14:08:40.238 DEBUG:teuthology.parallel:result is None 2024-08-19T14:08:40.239 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T14:08:40.392 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-19T14:08:40.413 INFO:teuthology.orchestra.run.smithi102.stdout:19.1.0-1382.gde09dd01.el9 2024-08-19T14:08:40.413 INFO:teuthology.packaging:The installed version of ceph is 19.1.0-1382.gde09dd01.el9 2024-08-19T14:08:40.413 INFO:teuthology.task.install:The correct ceph version 19.1.0-1382.gde09dd01 is installed. 2024-08-19T14:08:40.416 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-19T14:08:40.416 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-08-19T14:08:40.416 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-19T14:08:40.478 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-19T14:08:40.478 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-08-19T14:08:40.479 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/daemon-helper 2024-08-19T14:08:40.542 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-19T14:08:40.611 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-19T14:08:40.611 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-08-19T14:08:40.612 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-19T14:08:40.676 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-19T14:08:40.741 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-19T14:08:40.741 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-08-19T14:08:40.741 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/stdin-killer 2024-08-19T14:08:40.806 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-19T14:08:40.871 INFO:teuthology.run_tasks:Running task workunit... 2024-08-19T14:08:40.887 INFO:tasks.workunit:Pulling workunits from ref 78215f5a1cf5eaf321d506a66539942e12851f34 2024-08-19T14:08:40.888 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-19T14:08:40.888 INFO:tasks.workunit:timeout=3h 2024-08-19T14:08:40.888 INFO:tasks.workunit:cleanup=True 2024-08-19T14:08:40.889 DEBUG:teuthology.orchestra.run.smithi102:> stat -- /home/ubuntu/cephtest/mnt.0 2024-08-19T14:08:40.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:08:40.927 INFO:teuthology.orchestra.run.smithi102.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-19T14:08:40.927 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-08-19T14:08:40.982 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-08-19T14:08:40.982 DEBUG:teuthology.orchestra.run.smithi102:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-08-19T14:08:41.038 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 78215f5a1cf5eaf321d506a66539942e12851f34 2024-08-19T14:08:41.093 INFO:tasks.workunit.client.0.smithi102.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-08-19T14:09:33.794 INFO:tasks.workunit.client.0.smithi102.stderr:Note: switching to '78215f5a1cf5eaf321d506a66539942e12851f34'. 2024-08-19T14:09:33.795 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-08-19T14:09:33.795 INFO:tasks.workunit.client.0.smithi102.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-08-19T14:09:33.795 INFO:tasks.workunit.client.0.smithi102.stderr:changes and commit them, and you can discard any commits you make in this 2024-08-19T14:09:33.795 INFO:tasks.workunit.client.0.smithi102.stderr:state without impacting any branches by switching back to a branch. 2024-08-19T14:09:33.795 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-08-19T14:09:33.795 INFO:tasks.workunit.client.0.smithi102.stderr:If you want to create a new branch to retain commits you create, you may 2024-08-19T14:09:33.795 INFO:tasks.workunit.client.0.smithi102.stderr:do so (now or later) by using -c with the switch command. Example: 2024-08-19T14:09:33.796 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-08-19T14:09:33.796 INFO:tasks.workunit.client.0.smithi102.stderr: git switch -c 2024-08-19T14:09:33.796 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-08-19T14:09:33.796 INFO:tasks.workunit.client.0.smithi102.stderr:Or undo this operation with: 2024-08-19T14:09:33.796 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-08-19T14:09:33.796 INFO:tasks.workunit.client.0.smithi102.stderr: git switch - 2024-08-19T14:09:33.796 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-08-19T14:09:33.796 INFO:tasks.workunit.client.0.smithi102.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-08-19T14:09:33.796 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-08-19T14:09:33.796 INFO:tasks.workunit.client.0.smithi102.stderr:HEAD is now at 78215f5a1cf Merge pull request #59291 from zdover23/wip-doc-2024-08-19-backport-59290-to-squid 2024-08-19T14:09:33.799 DEBUG:teuthology.orchestra.run.smithi102:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-08-19T14:09:33.856 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-08-19T14:09:33.856 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-08-19T14:09:33.914 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2024-08-19T14:09:33.915 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2024-08-19T14:09:33.915 DEBUG:teuthology.orchestra.run.smithi102:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=78215f5a1cf5eaf321d506a66539942e12851f34 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2024-08-19T14:09:33.983 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-19T14:09:33.991 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-19T14:09:33.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-08-19T14:09:33.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-08-19T14:09:33.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-08-19T14:09:33.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-08-19T14:09:33.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-08-19T14:09:33.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-08-19T14:09:33.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-08-19T14:09:33.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-08-19T14:09:33.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-mapper 2024-08-19T14:09:33.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2024-08-19T14:09:33.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2024-08-19T14:09:33.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2024-08-19T14:09:33.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2024-08-19T14:09:33.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2024-08-19T14:09:33.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2024-08-19T14:09:34.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=dc43965e-692e-492a-b025-1c03b4666670 --auth-supported=none ' 2024-08-19T14:09:34.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-08-19T14:09:34.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-08-19T14:09:34.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2024-08-19T14:09:34.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-08-19T14:09:34.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2024-08-19T14:09:34.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2024-08-19T14:09:34.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2024-08-19T14:09:34.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-mapper 2024-08-19T14:09:34.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-mapper 2024-08-19T14:09:34.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-08-19T14:09:34.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:09:34.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-08-19T14:09:34.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:09:34.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:09:34.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:09:34.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:09:34.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:09:34.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:09:34.013 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:09:34.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:09:34.014 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:09:34.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:09:34.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:09:34.014 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:09:34.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:09:34.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:09:34.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:09:34.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:09:34.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:09:34.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:09:34.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-08-19T14:09:34.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:09:34.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:09:34.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.15855 2024-08-19T14:09:34.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.15855 2024-08-19T14:09:34.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:09:34.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:09:34.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-mapper 2024-08-19T14:09:34.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T14:09:34.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:09:34.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.15855 2024-08-19T14:09:34.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.15855 2024-08-19T14:09:34.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T14:09:34.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-19T14:09:34.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-19T14:09:34.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T14:09:34.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2024-08-19T14:09:34.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2024-08-19T14:09:34.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2024-08-19T14:09:34.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-08-19T14:09:34.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2024-08-19T14:09:34.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2024-08-19T14:09:34.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2024-08-19T14:09:34.026 INFO:tasks.workunit.client.0.smithi102.stdout:Dir: td/osd-mapper 2024-08-19T14:09:34.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2024-08-19T14:09:34.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2024-08-19T14:09:34.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-mapper 2024-08-19T14:09:34.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-08-19T14:09:34.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-08-19T14:09:34.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2024-08-19T14:09:34.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2024-08-19T14:09:34.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2024-08-19T14:09:34.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2024-08-19T14:09:34.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-08-19T14:09:34.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-08-19T14:09:34.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-08-19T14:09:35.654 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 3cab53a5-c985-4dbb-bc71-967899357988 2024-08-19T14:09:35.870 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:09:35.926 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:09:35.925+0000 7fe7ff80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:09:35.928 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:09:35.927+0000 7fe7ff80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:09:35.929 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:09:35.928+0000 7fe7ff80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:09:35.929 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:09:35.929+0000 7fe7ff80d640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-08-19T14:09:35.929 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:09:35.929+0000 7fe7ff80d640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-08-19T14:09:35.930 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:09:35.929+0000 7fe7ff80d640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-08-19T14:09:36.014 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:09:36.013+0000 7fe7ff80d640 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2024-08-19T14:09:41.361 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T14:09:41.698 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T14:09:41.730 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:09:41.726+0000 7f947cc1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:09:41.730 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:09:41.728+0000 7f947cc1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:09:41.730 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:09:41.729+0000 7f947cc1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:09:41.961 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:09:42.253 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:09:42.253+0000 7f947cc1a640 -1 Falling back to public interface 2024-08-19T14:09:43.223 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:09:44.491 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:09:44.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:09:44.578+0000 7f947cc1a640 -1 osd.0 0 log_to_monitors true 2024-08-19T14:09:45.765 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:09:47.038 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:09:48.341 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:09:49.612 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:09:50.885 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:09:52.159 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:09:53.466 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:09:54.738 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:09:56.012 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:09:57.859 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:09:59.349 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:10:00.998 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:10:03.016 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:10:04.342 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:10:05.808 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:10:07.271 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:10:09.231 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T14:10:09.547 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3989337452,v1:127.0.0.1:6803/3989337452] [v2:127.0.0.1:6804/3989337452,v1:127.0.0.1:6805/3989337452] exists,up 3cab53a5-c985-4dbb-bc71-967899357988 2024-08-19T14:10:09.551 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 4fb23cde-f6a0-4d54-81e5-e2b009ad78d2 2024-08-19T14:10:09.970 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:10:10.012 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:10.012+0000 7f8d06bcb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:10:10.030 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:10.029+0000 7f8d06bcb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:10:10.030 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:10.030+0000 7f8d06bcb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:10:10.031 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:10.030+0000 7f8d06bcb640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-08-19T14:10:10.031 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:10.030+0000 7f8d06bcb640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-08-19T14:10:10.031 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:10.030+0000 7f8d06bcb640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-08-19T14:10:10.096 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:10.096+0000 7f8d06bcb640 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2024-08-19T14:10:15.924 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T14:10:16.265 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T14:10:16.294 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:16.293+0000 7f969c988640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:10:16.295 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:16.295+0000 7f969c988640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:10:16.297 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:16.296+0000 7f969c988640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:10:16.530 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:10:16.818 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:16.818+0000 7f969c988640 -1 Falling back to public interface 2024-08-19T14:10:17.800 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:10:19.070 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:10:19.237 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:19.236+0000 7f969c988640 -1 osd.1 0 log_to_monitors true 2024-08-19T14:10:20.353 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:10:21.625 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:10:22.898 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:10:24.201 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:10:25.477 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:10:26.752 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:10:28.029 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:10:29.302 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:10:30.593 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:10:31.983 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:10:33.710 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:10:35.323 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:10:36.908 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:10:38.326 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:10:39.601 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:10:41.242 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:10:42.560 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T14:10:42.859 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/959681097,v1:127.0.0.1:6811/959681097] [v2:127.0.0.1:6812/959681097,v1:127.0.0.1:6813/959681097] exists,up 4fb23cde-f6a0-4d54-81e5-e2b009ad78d2 2024-08-19T14:10:42.863 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 228d2e03-bfd5-46c4-84e7-0cea44488181 2024-08-19T14:10:43.329 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:10:43.372 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:43.370+0000 7fcb7501a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:10:43.406 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:43.404+0000 7fcb7501a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:10:43.406 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:43.405+0000 7fcb7501a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:10:43.407 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:43.406+0000 7fcb7501a640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-08-19T14:10:43.407 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:43.406+0000 7fcb7501a640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-08-19T14:10:43.407 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:43.406+0000 7fcb7501a640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-08-19T14:10:43.481 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:43.479+0000 7fcb7501a640 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2024-08-19T14:10:49.207 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T14:10:49.544 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T14:10:49.574 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:49.574+0000 7fab8ae1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:10:49.576 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:49.576+0000 7fab8ae1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:10:49.577 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:49.577+0000 7fab8ae1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:10:49.813 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:10:50.100 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:50.100+0000 7fab8ae1e640 -1 Falling back to public interface 2024-08-19T14:10:51.077 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:10:52.349 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:10:52.431 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:10:52.431+0000 7fab8ae1e640 -1 osd.2 0 log_to_monitors true 2024-08-19T14:10:53.624 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:10:54.900 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:10:56.224 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:10:57.495 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:10:58.765 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:11:00.067 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:11:01.403 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:11:02.677 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:11:04.012 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:11:05.418 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:11:06.793 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:11:08.546 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:11:09.990 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:11:11.272 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:11:12.819 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:11:14.538 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:11:16.075 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T14:11:16.344 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4079983981,v1:127.0.0.1:6819/4079983981] [v2:127.0.0.1:6820/4079983981,v1:127.0.0.1:6821/4079983981] exists,up 228d2e03-bfd5-46c4-84e7-0cea44488181 2024-08-19T14:11:16.777 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' created 2024-08-19T14:11:18.794 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836498 2024-08-19T14:11:19.092 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672972 2024-08-19T14:11:21.904 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509447 2024-08-19T14:11:24.390 INFO:tasks.workunit.client.0.smithi102.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-08-19T14:11:24.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2024-08-19T14:11:24.484 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: { 2024-08-19T14:11:24.484 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-08-19T14:11:24.484 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:11:24.498 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: { 2024-08-19T14:11:24.498 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-08-19T14:11:24.498 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:11:24.510 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: { 2024-08-19T14:11:24.510 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-08-19T14:11:24.510 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:11:24.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2024-08-19T14:11:24.622 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: { 2024-08-19T14:11:24.622 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-08-19T14:11:24.622 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:11:24.636 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: { 2024-08-19T14:11:24.636 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-08-19T14:11:24.636 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:11:24.648 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: { 2024-08-19T14:11:24.649 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-08-19T14:11:24.649 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:11:24.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2024-08-19T14:11:24.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2024-08-19T14:11:24.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2024-08-19T14:11:24.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2024-08-19T14:11:24.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2024-08-19T14:11:24.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-08-19T14:11:24.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-08-19T14:11:25.052 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap01 2024-08-19T14:11:25.268 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap02 2024-08-19T14:11:25.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2024-08-19T14:11:25.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-08-19T14:11:25.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-08-19T14:11:25.664 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap13 2024-08-19T14:11:25.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2024-08-19T14:11:25.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-08-19T14:11:25.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-08-19T14:11:26.100 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap24 2024-08-19T14:11:26.308 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap25 2024-08-19T14:11:26.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 10272 2024-08-19T14:11:26.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2024-08-19T14:11:26.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-08-19T14:11:26.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2024-08-19T14:11:26.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2024-08-19T14:11:26.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-08-19T14:11:26.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2024-08-19T14:11:27.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2024-08-19T14:11:27.175 INFO:tasks.workunit.client.0.smithi102.stdout:pgid is 1.3 (primary: osd.1) 2024-08-19T14:11:27.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2024-08-19T14:11:27.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2024-08-19T14:11:27.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: local pgid=1.3 2024-08-19T14:11:27.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: ceph pg dump pgs_brief 2024-08-19T14:11:27.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-08-19T14:11:27.425 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs_brief 2024-08-19T14:11:27.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: local prim_osd=1 2024-08-19T14:11:27.437 INFO:tasks.workunit.client.0.smithi102.stdout:Setting scrub debug data. Primary for 1.3 is 1 2024-08-19T14:11:27.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2024-08-19T14:11:27.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: get_asok_path osd.1 2024-08-19T14:11:27.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T14:11:27.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T14:11:27.439 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:11:27.439 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:11:27.439 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.15855 2024-08-19T14:11:27.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.15855/ceph-osd.1.asok 2024-08-19T14:11:27.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: CEPH_ARGS= 2024-08-19T14:11:27.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.15855/ceph-osd.1.asok scrubdebug 1.3 set sessions 2024-08-19T14:11:27.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-08-19T14:11:27.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-08-19T14:11:27.552 INFO:tasks.workunit.client.0.smithi102.stdout:{"success":true}{ 2024-08-19T14:11:27.552 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "objxxx", 2024-08-19T14:11:27.553 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T14:11:27.553 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T14:11:27.553 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-08-19T14:11:27.553 INFO:tasks.workunit.client.0.smithi102.stdout: "snapshots": [ 2024-08-19T14:11:27.553 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T14:11:27.553 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-08-19T14:11:27.553 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "snap01" 2024-08-19T14:11:27.553 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T14:11:27.553 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T14:11:27.553 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-08-19T14:11:27.553 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "snap02" 2024-08-19T14:11:27.553 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T14:11:27.553 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T14:11:27.554 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 6, 2024-08-19T14:11:27.554 INFO:tasks.workunit.client.0.smithi102.stdout: "overlaps": [] 2024-08-19T14:11:27.554 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T14:11:27.554 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T14:11:27.554 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 3, 2024-08-19T14:11:27.554 INFO:tasks.workunit.client.0.smithi102.stdout: "snapshots": [ 2024-08-19T14:11:27.554 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T14:11:27.554 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 3, 2024-08-19T14:11:27.554 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "snap13" 2024-08-19T14:11:27.554 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T14:11:27.554 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T14:11:27.554 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 6, 2024-08-19T14:11:27.554 INFO:tasks.workunit.client.0.smithi102.stdout: "overlaps": [] 2024-08-19T14:11:27.554 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T14:11:27.555 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T14:11:27.555 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 5, 2024-08-19T14:11:27.555 INFO:tasks.workunit.client.0.smithi102.stdout: "snapshots": [ 2024-08-19T14:11:27.555 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T14:11:27.555 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 4, 2024-08-19T14:11:27.555 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "snap24" 2024-08-19T14:11:27.555 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T14:11:27.555 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T14:11:27.555 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 5, 2024-08-19T14:11:27.555 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "snap25" 2024-08-19T14:11:27.555 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T14:11:27.555 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T14:11:27.556 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 5, 2024-08-19T14:11:27.556 INFO:tasks.workunit.client.0.smithi102.stdout: "overlaps": [] 2024-08-19T14:11:27.556 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T14:11:27.556 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T14:11:27.556 INFO:tasks.workunit.client.0.smithi102.stdout: "id": "head", 2024-08-19T14:11:27.556 INFO:tasks.workunit.client.0.smithi102.stdout: "snapshots": [], 2024-08-19T14:11:27.556 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 6 2024-08-19T14:11:27.556 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T14:11:27.556 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T14:11:27.556 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:11:27.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-08-19T14:11:27.646 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T14:11:27.647 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": true, 2024-08-19T14:11:27.647 INFO:tasks.workunit.client.0.smithi102.stdout: "must": true, 2024-08-19T14:11:27.647 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "0.000000" 2024-08-19T14:11:27.647 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:11:27.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2024-08-19T14:11:30.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2024-08-19T14:11:30.915 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T14:11:30.915 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-08-19T14:11:18.170926+0000 26'7 26:34 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-19T14:11:16.685013+0000 0'0 2024-08-19T14:11:16.685013+0000 0 0 queued for deep scrub 0 0 2024-08-19T14:11:30.915 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:11:18.170827+0000 0'0 26:31 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-19T14:11:16.685013+0000 0'0 2024-08-19T14:11:16.685013+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:11:16.685013+0000 0 0 2024-08-19T14:11:30.915 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:11:17.659064+0000 0'0 24:25 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-19T14:11:16.685013+0000 0'0 2024-08-19T14:11:16.685013+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:11:16.685013+0000 0 0 2024-08-19T14:11:30.915 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:11:18.170820+0000 0'0 26:28 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-19T14:11:16.685013+0000 0'0 2024-08-19T14:11:16.685013+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:11:16.685013+0000 0 0 2024-08-19T14:11:30.915 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:30.915 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T14:11:30.916 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:11:30.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2024-08-19T14:11:30.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2024-08-19T14:11:31.173 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T14:11:31.174 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-08-19T14:11:18.170926+0000 26'7 26:34 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-19T14:11:16.685013+0000 0'0 2024-08-19T14:11:16.685013+0000 0 0 queued for deep scrub 0 0 2024-08-19T14:11:31.174 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:11:18.170827+0000 0'0 26:31 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-19T14:11:16.685013+0000 0'0 2024-08-19T14:11:16.685013+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:11:16.685013+0000 0 0 2024-08-19T14:11:31.174 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:11:17.659064+0000 0'0 24:25 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-19T14:11:16.685013+0000 0'0 2024-08-19T14:11:16.685013+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:11:16.685013+0000 0 0 2024-08-19T14:11:31.174 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:11:18.170820+0000 0'0 26:28 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-19T14:11:16.685013+0000 0'0 2024-08-19T14:11:16.685013+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:11:16.685013+0000 0 0 2024-08-19T14:11:31.174 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:31.174 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T14:11:31.174 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:11:31.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2024-08-19T14:11:31.541 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T14:11:31.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2024-08-19T14:11:32.060 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T14:11:32.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2024-08-19T14:11:37.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2024-08-19T14:11:37.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-08-19T14:11:37.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:11:37.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:11:37.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:11:37.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:11:37.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:11:37.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:11:37.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-08-19T14:11:37.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-08-19T14:11:37.410 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.409+0000 7f9793320980 1 bdev(0x55e1fcd06000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-08-19T14:11:37.410 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.409+0000 7f9793320980 0 bdev(0x55e1fcd06000 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-08-19T14:11:37.410 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.409+0000 7f9793320980 1 bdev(0x55e1fcd06000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-08-19T14:11:37.410 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.409+0000 7f9793320980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-08-19T14:11:37.410 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.409+0000 7f9793320980 1 bdev(0x55e1fcd06700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-08-19T14:11:37.410 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.409+0000 7f9793320980 0 bdev(0x55e1fcd06700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-08-19T14:11:37.410 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.409+0000 7f9793320980 1 bdev(0x55e1fcd06700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-08-19T14:11:37.410 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.409+0000 7f9793320980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-08-19T14:11:37.410 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.409+0000 7f9793320980 1 bluefs mount 2024-08-19T14:11:37.424 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.424+0000 7f9793320980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-08-19T14:11:37.450 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.449+0000 7f9793320980 1 bluefs mount shared_bdev_used = 0 2024-08-19T14:11:37.450 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.450+0000 7f9793320980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-08-19T14:11:37.453 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.452+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-08-19T14:11:37.453 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.452+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-08-19T14:11:37.453 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.452+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-08-19T14:11:37.454 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.452+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-08-19T14:11:37.454 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.452+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-08-19T14:11:37.454 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.452+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-08-19T14:11:37.454 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.452+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-08-19T14:11:37.454 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.453+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-08-19T14:11:37.454 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.453+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-08-19T14:11:37.454 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.453+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-08-19T14:11:37.454 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.453+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-08-19T14:11:37.454 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.453+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-08-19T14:11:37.488 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.487+0000 7f9793320980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-08-19T14:11:37.488 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.487+0000 7f9793320980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-08-19T14:11:37.488 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.487+0000 7f9793320980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-08-19T14:11:37.488 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.487+0000 7f9793320980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-08-19T14:11:37.488 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.487+0000 7f9793320980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-08-19T14:11:37.489 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.487+0000 7f9793320980 1 freelist init 2024-08-19T14:11:37.489 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.487+0000 7f9793320980 1 freelist _read_cfg 2024-08-19T14:11:37.489 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.488+0000 7f9793320980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 24 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 8.77383e-07 2024-08-19T14:11:37.489 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.488+0000 7f9793320980 1 bluefs umount 2024-08-19T14:11:37.489 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.488+0000 7f9793320980 1 bdev(0x55e1fcd06700 td/osd-mapper/0/block) close 2024-08-19T14:11:37.671 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.671+0000 7f9793320980 1 bdev(0x55e1fcd06700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-08-19T14:11:37.671 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.671+0000 7f9793320980 0 bdev(0x55e1fcd06700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-08-19T14:11:37.672 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.671+0000 7f9793320980 1 bdev(0x55e1fcd06700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-08-19T14:11:37.672 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.671+0000 7f9793320980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-08-19T14:11:37.672 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.671+0000 7f9793320980 1 bluefs mount 2024-08-19T14:11:37.672 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.671+0000 7f9793320980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-08-19T14:11:37.678 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.678+0000 7f9793320980 1 bluefs mount shared_bdev_used = 27459584 2024-08-19T14:11:37.678 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.678+0000 7f9793320980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-08-19T14:11:37.681 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.680+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-08-19T14:11:37.681 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.680+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-08-19T14:11:37.681 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.680+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-08-19T14:11:37.682 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.680+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-08-19T14:11:37.682 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.680+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-08-19T14:11:37.682 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.680+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-08-19T14:11:37.682 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.680+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-08-19T14:11:37.682 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.680+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-08-19T14:11:37.682 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.680+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-08-19T14:11:37.682 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.680+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-08-19T14:11:37.682 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.680+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-08-19T14:11:37.682 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:37.680+0000 7f9793320980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-08-19T14:11:38.361 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.360+0000 7f9793320980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-08-19T14:11:38.363 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-08-19T14:11:38.363 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 0b 05 58 02 00 00 dc 43 96 5e 69 2e 49 2a b0 25 |..X....C.^i.I*.%| 2024-08-19T14:11:38.363 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 1c 03 b4 66 66 70 00 00 00 00 1c 00 00 00 00 00 |...ffp..........| 2024-08-19T14:11:38.363 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-19T14:11:38.363 INFO:tasks.workunit.client.0.smithi102.stdout:* 2024-08-19T14:11:38.363 INFO:tasks.workunit.client.0.smithi102.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-08-19T14:11:38.363 INFO:tasks.workunit.client.0.smithi102.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-08-19T14:11:38.363 INFO:tasks.workunit.client.0.smithi102.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-08-19T14:11:38.363 INFO:tasks.workunit.client.0.smithi102.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-08-19T14:11:38.363 INFO:tasks.workunit.client.0.smithi102.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-08-19T14:11:38.364 INFO:tasks.workunit.client.0.smithi102.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-08-19T14:11:38.364 INFO:tasks.workunit.client.0.smithi102.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-08-19T14:11:38.364 INFO:tasks.workunit.client.0.smithi102.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-08-19T14:11:38.364 INFO:tasks.workunit.client.0.smithi102.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-08-19T14:11:38.364 INFO:tasks.workunit.client.0.smithi102.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-08-19T14:11:38.364 INFO:tasks.workunit.client.0.smithi102.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-08-19T14:11:38.364 INFO:tasks.workunit.client.0.smithi102.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-08-19T14:11:38.364 INFO:tasks.workunit.client.0.smithi102.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-08-19T14:11:38.364 INFO:tasks.workunit.client.0.smithi102.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-08-19T14:11:38.364 INFO:tasks.workunit.client.0.smithi102.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-08-19T14:11:38.364 INFO:tasks.workunit.client.0.smithi102.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-08-19T14:11:38.364 INFO:tasks.workunit.client.0.smithi102.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-08-19T14:11:38.365 INFO:tasks.workunit.client.0.smithi102.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-08-19T14:11:38.365 INFO:tasks.workunit.client.0.smithi102.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-08-19T14:11:38.365 INFO:tasks.workunit.client.0.smithi102.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-08-19T14:11:38.365 INFO:tasks.workunit.client.0.smithi102.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-08-19T14:11:38.365 INFO:tasks.workunit.client.0.smithi102.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-08-19T14:11:38.365 INFO:tasks.workunit.client.0.smithi102.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-08-19T14:11:38.365 INFO:tasks.workunit.client.0.smithi102.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-08-19T14:11:38.365 INFO:tasks.workunit.client.0.smithi102.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-08-19T14:11:38.365 INFO:tasks.workunit.client.0.smithi102.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-08-19T14:11:38.365 INFO:tasks.workunit.client.0.smithi102.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-08-19T14:11:38.365 INFO:tasks.workunit.client.0.smithi102.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-08-19T14:11:38.365 INFO:tasks.workunit.client.0.smithi102.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-08-19T14:11:38.365 INFO:tasks.workunit.client.0.smithi102.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-08-19T14:11:38.366 INFO:tasks.workunit.client.0.smithi102.stdout:00000220 72 65 1c 00 00 00 05 00 00 00 3c ab 53 a5 c9 85 |re........<.S...| 2024-08-19T14:11:38.366 INFO:tasks.workunit.client.0.smithi102.stdout:00000230 4d bb bc 71 96 78 99 35 79 88 00 00 00 00 00 00 |M..q.x.5y.......| 2024-08-19T14:11:38.366 INFO:tasks.workunit.client.0.smithi102.stdout:00000240 00 00 1c 00 00 00 3e 52 c3 66 e5 68 5a 2c 01 00 |......>R.f.hZ,..| 2024-08-19T14:11:38.366 INFO:tasks.workunit.client.0.smithi102.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-08-19T14:11:38.366 INFO:tasks.workunit.client.0.smithi102.stdout:0000025e 2024-08-19T14:11:38.366 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:38.366 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-08-19T14:11:38.366 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:38.366 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-08-19T14:11:38.366 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-08-19T14:11:38.366 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-08-19T14:11:38.366 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-08-19T14:11:38.367 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-08-19T14:11:38.367 INFO:tasks.workunit.client.0.smithi102.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-08-19T14:11:38.367 INFO:tasks.workunit.client.0.smithi102.stdout:00000047 2024-08-19T14:11:38.367 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:38.367 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:38.367 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-08-19T14:11:38.367 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-08-19T14:11:38.367 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-08-19T14:11:38.367 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-08-19T14:11:38.367 INFO:tasks.workunit.client.0.smithi102.stdout:0000003f 2024-08-19T14:11:38.367 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:38.367 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-08-19T14:11:38.367 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-08-19T14:11:38.368 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-08-19T14:11:38.368 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-08-19T14:11:38.368 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-08-19T14:11:38.368 INFO:tasks.workunit.client.0.smithi102.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-08-19T14:11:38.368 INFO:tasks.workunit.client.0.smithi102.stdout:00000047 2024-08-19T14:11:38.368 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:38.368 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-08-19T14:11:38.368 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:38.368 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:38.368 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:38.368 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:38.368 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:38.368 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:38.369 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-08-19T14:11:38.369 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:38.369 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:38.369 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:38.369 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:38.369 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:38.369 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:38.369 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:38.369 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:38.369 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:38.369 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:38.369 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:38.369 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:38.369 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:38.370 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-08-19T14:11:38.370 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:38.370 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:38.370 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:38.370 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:38.370 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:38.370 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:38.370 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-08-19T14:11:38.370 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:38.370 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:38.370 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:38.370 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:38.370 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:38.371 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:38.371 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-08-19T14:11:38.371 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:38.371 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-08-19T14:11:38.371 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:38.371 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.361+0000 7f9793320980 1 bluefs umount 2024-08-19T14:11:38.371 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.361+0000 7f9793320980 1 bdev(0x55e1fcd06700 td/osd-mapper/0/block) close 2024-08-19T14:11:38.607 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.607+0000 7f9793320980 1 freelist shutdown 2024-08-19T14:11:38.607 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.607+0000 7f9793320980 1 bdev(0x55e1fcd06000 td/osd-mapper/0/block) close 2024-08-19T14:11:38.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-08-19T14:11:38.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-08-19T14:11:38.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2024-08-19T14:11:38.703 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.701+0000 7fa4c9d18980 1 bdev(0x55a045d32000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-08-19T14:11:38.703 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.701+0000 7fa4c9d18980 0 bdev(0x55a045d32000 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-08-19T14:11:38.703 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.702+0000 7fa4c9d18980 1 bdev(0x55a045d32000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-08-19T14:11:38.703 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.702+0000 7fa4c9d18980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-08-19T14:11:38.703 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.702+0000 7fa4c9d18980 1 bdev(0x55a045d32700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-08-19T14:11:38.703 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.702+0000 7fa4c9d18980 0 bdev(0x55a045d32700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-08-19T14:11:38.703 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.702+0000 7fa4c9d18980 1 bdev(0x55a045d32700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-08-19T14:11:38.703 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.702+0000 7fa4c9d18980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-08-19T14:11:38.703 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.702+0000 7fa4c9d18980 1 bluefs mount 2024-08-19T14:11:38.713 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.713+0000 7fa4c9d18980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-08-19T14:11:38.738 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.738+0000 7fa4c9d18980 1 bluefs mount shared_bdev_used = 0 2024-08-19T14:11:38.739 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.738+0000 7fa4c9d18980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-08-19T14:11:38.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.740+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-08-19T14:11:38.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.740+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-08-19T14:11:38.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.740+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-08-19T14:11:38.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.740+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-08-19T14:11:38.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.740+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-08-19T14:11:38.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.741+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-08-19T14:11:38.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.741+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-08-19T14:11:38.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.741+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-08-19T14:11:38.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.741+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-08-19T14:11:38.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.741+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-08-19T14:11:38.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.741+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-08-19T14:11:38.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.741+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-08-19T14:11:38.773 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.773+0000 7fa4c9d18980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-08-19T14:11:38.774 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.773+0000 7fa4c9d18980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-08-19T14:11:38.774 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.773+0000 7fa4c9d18980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-08-19T14:11:38.774 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.773+0000 7fa4c9d18980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-08-19T14:11:38.774 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.773+0000 7fa4c9d18980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-08-19T14:11:38.774 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.773+0000 7fa4c9d18980 1 freelist init 2024-08-19T14:11:38.774 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.773+0000 7fa4c9d18980 1 freelist _read_cfg 2024-08-19T14:11:38.775 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.774+0000 7fa4c9d18980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 14 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 4.95912e-07 2024-08-19T14:11:38.775 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.774+0000 7fa4c9d18980 1 bluefs umount 2024-08-19T14:11:38.775 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.774+0000 7fa4c9d18980 1 bdev(0x55a045d32700 td/osd-mapper/2/block) close 2024-08-19T14:11:38.964 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.964+0000 7fa4c9d18980 1 bdev(0x55a045d32700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-08-19T14:11:38.965 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.964+0000 7fa4c9d18980 0 bdev(0x55a045d32700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-08-19T14:11:38.965 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.964+0000 7fa4c9d18980 1 bdev(0x55a045d32700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-08-19T14:11:38.965 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.964+0000 7fa4c9d18980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-08-19T14:11:38.965 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.964+0000 7fa4c9d18980 1 bluefs mount 2024-08-19T14:11:38.965 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.964+0000 7fa4c9d18980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-08-19T14:11:38.971 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.970+0000 7fa4c9d18980 1 bluefs mount shared_bdev_used = 27459584 2024-08-19T14:11:38.971 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.970+0000 7fa4c9d18980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-08-19T14:11:38.974 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.972+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-08-19T14:11:38.974 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.973+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-08-19T14:11:38.974 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.973+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-08-19T14:11:38.974 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.973+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-08-19T14:11:38.974 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.973+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-08-19T14:11:38.974 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.973+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-08-19T14:11:38.974 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.973+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-08-19T14:11:38.975 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.973+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-08-19T14:11:38.975 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.973+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-08-19T14:11:38.975 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.973+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-08-19T14:11:38.975 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.973+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-08-19T14:11:38.975 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:38.973+0000 7fa4c9d18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-08-19T14:11:39.924 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:39.923+0000 7fa4c9d18980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-08-19T14:11:39.924 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:39.924+0000 7fa4c9d18980 1 bluefs umount 2024-08-19T14:11:39.924 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:39.924+0000 7fa4c9d18980 1 bdev(0x55a045d32700 td/osd-mapper/2/block) close 2024-08-19T14:11:40.160 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:40.160+0000 7fa4c9d18980 1 freelist shutdown 2024-08-19T14:11:40.160 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:40.160+0000 7fa4c9d18980 1 bdev(0x55a045d32000 td/osd-mapper/2/block) close 2024-08-19T14:11:40.218 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-08-19T14:11:40.218 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-08-19T14:11:40.218 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:40.218 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-08-19T14:11:40.218 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-08-19T14:11:40.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-08-19T14:11:40.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2024-08-19T14:11:40.219 INFO:tasks.workunit.client.0.smithi102.stderr:grep: /tmp/oo2.dump: No such file or directory 2024-08-19T14:11:40.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-08-19T14:11:40.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-08-19T14:11:40.927 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-08-19T14:11:40.928 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 0b 05 58 02 00 00 dc 43 96 5e 69 2e 49 2a b0 25 |..X....C.^i.I*.%| 2024-08-19T14:11:40.928 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 1c 03 b4 66 66 70 02 00 00 00 1c 00 00 00 00 00 |...ffp..........| 2024-08-19T14:11:40.928 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-19T14:11:40.928 INFO:tasks.workunit.client.0.smithi102.stdout:* 2024-08-19T14:11:40.928 INFO:tasks.workunit.client.0.smithi102.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-08-19T14:11:40.928 INFO:tasks.workunit.client.0.smithi102.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-08-19T14:11:40.928 INFO:tasks.workunit.client.0.smithi102.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-08-19T14:11:40.928 INFO:tasks.workunit.client.0.smithi102.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-08-19T14:11:40.928 INFO:tasks.workunit.client.0.smithi102.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-08-19T14:11:40.928 INFO:tasks.workunit.client.0.smithi102.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-08-19T14:11:40.929 INFO:tasks.workunit.client.0.smithi102.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-08-19T14:11:40.929 INFO:tasks.workunit.client.0.smithi102.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-08-19T14:11:40.929 INFO:tasks.workunit.client.0.smithi102.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-08-19T14:11:40.929 INFO:tasks.workunit.client.0.smithi102.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-08-19T14:11:40.929 INFO:tasks.workunit.client.0.smithi102.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-08-19T14:11:40.929 INFO:tasks.workunit.client.0.smithi102.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-08-19T14:11:40.929 INFO:tasks.workunit.client.0.smithi102.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-08-19T14:11:40.929 INFO:tasks.workunit.client.0.smithi102.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-08-19T14:11:40.929 INFO:tasks.workunit.client.0.smithi102.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-08-19T14:11:40.929 INFO:tasks.workunit.client.0.smithi102.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-08-19T14:11:40.929 INFO:tasks.workunit.client.0.smithi102.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-08-19T14:11:40.930 INFO:tasks.workunit.client.0.smithi102.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-08-19T14:11:40.930 INFO:tasks.workunit.client.0.smithi102.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-08-19T14:11:40.930 INFO:tasks.workunit.client.0.smithi102.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-08-19T14:11:40.930 INFO:tasks.workunit.client.0.smithi102.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-08-19T14:11:40.930 INFO:tasks.workunit.client.0.smithi102.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-08-19T14:11:40.930 INFO:tasks.workunit.client.0.smithi102.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-08-19T14:11:40.930 INFO:tasks.workunit.client.0.smithi102.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-08-19T14:11:40.930 INFO:tasks.workunit.client.0.smithi102.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-08-19T14:11:40.930 INFO:tasks.workunit.client.0.smithi102.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-08-19T14:11:40.930 INFO:tasks.workunit.client.0.smithi102.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-08-19T14:11:40.930 INFO:tasks.workunit.client.0.smithi102.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-08-19T14:11:40.931 INFO:tasks.workunit.client.0.smithi102.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-08-19T14:11:40.931 INFO:tasks.workunit.client.0.smithi102.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-08-19T14:11:40.931 INFO:tasks.workunit.client.0.smithi102.stdout:00000220 72 65 1c 00 00 00 0f 00 00 00 22 8d 2e 03 bf d5 |re........".....| 2024-08-19T14:11:40.931 INFO:tasks.workunit.client.0.smithi102.stdout:00000230 46 c4 84 e7 0c ea 44 48 81 81 00 00 00 00 00 00 |F.....DH........| 2024-08-19T14:11:40.931 INFO:tasks.workunit.client.0.smithi102.stdout:00000240 00 00 1c 00 00 00 83 52 c3 66 bf 9d c2 09 01 00 |.......R.f......| 2024-08-19T14:11:40.931 INFO:tasks.workunit.client.0.smithi102.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-08-19T14:11:40.931 INFO:tasks.workunit.client.0.smithi102.stdout:0000025e 2024-08-19T14:11:40.931 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:40.931 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-08-19T14:11:40.931 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:40.931 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-08-19T14:11:40.931 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-08-19T14:11:40.932 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-08-19T14:11:40.932 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-08-19T14:11:40.932 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-08-19T14:11:40.932 INFO:tasks.workunit.client.0.smithi102.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-08-19T14:11:40.932 INFO:tasks.workunit.client.0.smithi102.stdout:00000047 2024-08-19T14:11:40.932 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:40.932 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:40.932 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-08-19T14:11:40.932 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-08-19T14:11:40.932 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-08-19T14:11:40.932 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-08-19T14:11:40.932 INFO:tasks.workunit.client.0.smithi102.stdout:0000003f 2024-08-19T14:11:40.933 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:40.933 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-08-19T14:11:40.933 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-08-19T14:11:40.933 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-08-19T14:11:40.933 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-08-19T14:11:40.933 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-08-19T14:11:40.933 INFO:tasks.workunit.client.0.smithi102.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-08-19T14:11:40.933 INFO:tasks.workunit.client.0.smithi102.stdout:00000047 2024-08-19T14:11:40.933 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:40.933 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-08-19T14:11:40.933 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:40.934 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:40.934 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:40.934 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:40.934 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:40.934 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:40.934 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-08-19T14:11:40.934 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:40.934 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:40.934 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:40.934 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:40.934 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:40.934 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:40.935 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:40.935 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:40.935 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:40.935 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:40.935 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:40.935 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:40.935 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:40.935 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-08-19T14:11:40.935 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:40.935 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:40.935 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:40.935 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:40.936 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:40.936 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:40.936 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-08-19T14:11:40.936 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:40.936 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:40.936 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:40.936 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:40.936 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:40.936 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:40.936 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-08-19T14:11:40.936 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:40.936 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-08-19T14:11:40.937 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:41.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-08-19T14:11:41.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-08-19T14:11:41.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-08-19T14:11:41.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2024-08-19T14:11:42.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2024-08-19T14:11:42.560 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2024-08-19T14:11:42.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2024-08-19T14:11:42.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-08-19T14:11:42.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2024-08-19T14:11:42.562 INFO:tasks.workunit.client.0.smithi102.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2024-08-19T14:11:42.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2024-08-19T14:11:42.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-08-19T14:11:42.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-08-19T14:11:42.587 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.587+0000 7fec33a26980 1 bdev(0x55b1af716000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-08-19T14:11:42.588 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.587+0000 7fec33a26980 0 bdev(0x55b1af716000 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-08-19T14:11:42.588 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.587+0000 7fec33a26980 1 bdev(0x55b1af716000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-08-19T14:11:42.588 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.587+0000 7fec33a26980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-08-19T14:11:42.588 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.587+0000 7fec33a26980 1 bdev(0x55b1af716700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-08-19T14:11:42.588 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.587+0000 7fec33a26980 0 bdev(0x55b1af716700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-08-19T14:11:42.588 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.587+0000 7fec33a26980 1 bdev(0x55b1af716700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-08-19T14:11:42.588 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.587+0000 7fec33a26980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-08-19T14:11:42.588 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.587+0000 7fec33a26980 1 bluefs mount 2024-08-19T14:11:42.599 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.598+0000 7fec33a26980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-08-19T14:11:42.609 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.608+0000 7fec33a26980 1 bluefs mount shared_bdev_used = 0 2024-08-19T14:11:42.609 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.608+0000 7fec33a26980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-08-19T14:11:42.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.610+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-08-19T14:11:42.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.610+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-08-19T14:11:42.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.611+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-08-19T14:11:42.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.611+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-08-19T14:11:42.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.611+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-08-19T14:11:42.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.611+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-08-19T14:11:42.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.611+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-08-19T14:11:42.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.611+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-08-19T14:11:42.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.611+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-08-19T14:11:42.613 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.611+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-08-19T14:11:42.613 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.611+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-08-19T14:11:42.613 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.611+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-08-19T14:11:42.618 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.617+0000 7fec33a26980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-08-19T14:11:42.618 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.617+0000 7fec33a26980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-08-19T14:11:42.618 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.617+0000 7fec33a26980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-08-19T14:11:42.618 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.617+0000 7fec33a26980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-08-19T14:11:42.618 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.617+0000 7fec33a26980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-08-19T14:11:42.619 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.618+0000 7fec33a26980 1 freelist init 2024-08-19T14:11:42.619 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.618+0000 7fec33a26980 1 freelist _read_cfg 2024-08-19T14:11:42.619 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.618+0000 7fec33a26980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 24 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 8.77383e-07 2024-08-19T14:11:42.619 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.618+0000 7fec33a26980 1 bluefs umount 2024-08-19T14:11:42.619 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.618+0000 7fec33a26980 1 bdev(0x55b1af716700 td/osd-mapper/0/block) close 2024-08-19T14:11:42.853 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.853+0000 7fec33a26980 1 bdev(0x55b1af716700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-08-19T14:11:42.854 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.853+0000 7fec33a26980 0 bdev(0x55b1af716700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-08-19T14:11:42.854 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.853+0000 7fec33a26980 1 bdev(0x55b1af716700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-08-19T14:11:42.854 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.853+0000 7fec33a26980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-08-19T14:11:42.854 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.853+0000 7fec33a26980 1 bluefs mount 2024-08-19T14:11:42.857 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.857+0000 7fec33a26980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-08-19T14:11:42.866 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.865+0000 7fec33a26980 1 bluefs mount shared_bdev_used = 9568256 2024-08-19T14:11:42.866 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.865+0000 7fec33a26980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-08-19T14:11:42.869 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.867+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-08-19T14:11:42.869 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.868+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-08-19T14:11:42.869 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.868+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-08-19T14:11:42.869 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.868+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-08-19T14:11:42.869 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.868+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-08-19T14:11:42.869 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.868+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-08-19T14:11:42.869 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.868+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-08-19T14:11:42.869 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.868+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-08-19T14:11:42.869 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.868+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-08-19T14:11:42.869 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.868+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-08-19T14:11:42.870 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.868+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-08-19T14:11:42.870 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:42.868+0000 7fec33a26980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-08-19T14:11:43.123 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:43.122+0000 7fec33a26980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-08-19T14:11:43.125 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-08-19T14:11:43.125 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 0b 05 58 02 00 00 dc 43 96 5e 69 2e 49 2a b0 25 |..X....C.^i.I*.%| 2024-08-19T14:11:43.125 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 1c 03 b4 66 66 70 00 00 00 00 1c 00 00 00 00 00 |...ffp..........| 2024-08-19T14:11:43.125 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-19T14:11:43.125 INFO:tasks.workunit.client.0.smithi102.stdout:* 2024-08-19T14:11:43.125 INFO:tasks.workunit.client.0.smithi102.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-08-19T14:11:43.125 INFO:tasks.workunit.client.0.smithi102.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-08-19T14:11:43.125 INFO:tasks.workunit.client.0.smithi102.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-08-19T14:11:43.125 INFO:tasks.workunit.client.0.smithi102.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-08-19T14:11:43.125 INFO:tasks.workunit.client.0.smithi102.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-08-19T14:11:43.125 INFO:tasks.workunit.client.0.smithi102.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-08-19T14:11:43.125 INFO:tasks.workunit.client.0.smithi102.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-08-19T14:11:43.126 INFO:tasks.workunit.client.0.smithi102.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-08-19T14:11:43.126 INFO:tasks.workunit.client.0.smithi102.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-08-19T14:11:43.126 INFO:tasks.workunit.client.0.smithi102.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-08-19T14:11:43.126 INFO:tasks.workunit.client.0.smithi102.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-08-19T14:11:43.126 INFO:tasks.workunit.client.0.smithi102.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-08-19T14:11:43.126 INFO:tasks.workunit.client.0.smithi102.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-08-19T14:11:43.126 INFO:tasks.workunit.client.0.smithi102.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-08-19T14:11:43.126 INFO:tasks.workunit.client.0.smithi102.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-08-19T14:11:43.126 INFO:tasks.workunit.client.0.smithi102.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-08-19T14:11:43.126 INFO:tasks.workunit.client.0.smithi102.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-08-19T14:11:43.126 INFO:tasks.workunit.client.0.smithi102.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-08-19T14:11:43.126 INFO:tasks.workunit.client.0.smithi102.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-08-19T14:11:43.127 INFO:tasks.workunit.client.0.smithi102.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-08-19T14:11:43.127 INFO:tasks.workunit.client.0.smithi102.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-08-19T14:11:43.127 INFO:tasks.workunit.client.0.smithi102.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-08-19T14:11:43.127 INFO:tasks.workunit.client.0.smithi102.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-08-19T14:11:43.127 INFO:tasks.workunit.client.0.smithi102.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-08-19T14:11:43.127 INFO:tasks.workunit.client.0.smithi102.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-08-19T14:11:43.127 INFO:tasks.workunit.client.0.smithi102.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-08-19T14:11:43.127 INFO:tasks.workunit.client.0.smithi102.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-08-19T14:11:43.127 INFO:tasks.workunit.client.0.smithi102.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-08-19T14:11:43.127 INFO:tasks.workunit.client.0.smithi102.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-08-19T14:11:43.127 INFO:tasks.workunit.client.0.smithi102.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-08-19T14:11:43.127 INFO:tasks.workunit.client.0.smithi102.stdout:00000220 72 65 1c 00 00 00 05 00 00 00 3c ab 53 a5 c9 85 |re........<.S...| 2024-08-19T14:11:43.127 INFO:tasks.workunit.client.0.smithi102.stdout:00000230 4d bb bc 71 96 78 99 35 79 88 00 00 00 00 00 00 |M..q.x.5y.......| 2024-08-19T14:11:43.127 INFO:tasks.workunit.client.0.smithi102.stdout:00000240 00 00 1c 00 00 00 3e 52 c3 66 e5 68 5a 2c 01 00 |......>R.f.hZ,..| 2024-08-19T14:11:43.127 INFO:tasks.workunit.client.0.smithi102.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-08-19T14:11:43.128 INFO:tasks.workunit.client.0.smithi102.stdout:0000025e 2024-08-19T14:11:43.128 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:43.128 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-08-19T14:11:43.128 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:43.128 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-08-19T14:11:43.128 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-08-19T14:11:43.128 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-08-19T14:11:43.128 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-08-19T14:11:43.128 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-08-19T14:11:43.128 INFO:tasks.workunit.client.0.smithi102.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-08-19T14:11:43.128 INFO:tasks.workunit.client.0.smithi102.stdout:00000047 2024-08-19T14:11:43.128 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:43.128 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:43.129 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-08-19T14:11:43.129 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-08-19T14:11:43.129 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-08-19T14:11:43.129 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-08-19T14:11:43.129 INFO:tasks.workunit.client.0.smithi102.stdout:0000003f 2024-08-19T14:11:43.129 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:43.129 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-08-19T14:11:43.129 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-08-19T14:11:43.129 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-08-19T14:11:43.129 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-08-19T14:11:43.129 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-08-19T14:11:43.129 INFO:tasks.workunit.client.0.smithi102.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-08-19T14:11:43.130 INFO:tasks.workunit.client.0.smithi102.stdout:00000047 2024-08-19T14:11:43.130 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:43.130 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-08-19T14:11:43.130 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:43.130 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:43.130 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:43.130 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:43.130 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:43.130 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:43.130 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-08-19T14:11:43.130 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:43.130 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:43.130 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:43.131 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:43.131 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:43.131 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:43.131 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:43.131 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:43.131 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:43.131 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:43.131 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:43.132 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:43.132 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:43.132 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-08-19T14:11:43.132 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:43.132 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:43.132 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:43.132 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:43.132 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:43.132 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:43.132 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-08-19T14:11:43.132 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:43.132 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:43.132 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:43.133 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:43.133 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:43.133 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:43.133 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-08-19T14:11:43.133 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:43.133 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-08-19T14:11:43.133 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:43.133 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:43.123+0000 7fec33a26980 1 bluefs umount 2024-08-19T14:11:43.133 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:43.123+0000 7fec33a26980 1 bdev(0x55b1af716700 td/osd-mapper/0/block) close 2024-08-19T14:11:43.359 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:43.359+0000 7fec33a26980 1 freelist shutdown 2024-08-19T14:11:43.359 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:43.359+0000 7fec33a26980 1 bdev(0x55b1af716000 td/osd-mapper/0/block) close 2024-08-19T14:11:43.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-08-19T14:11:43.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-08-19T14:11:43.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-08-19T14:11:44.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:44.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-08-19T14:11:44.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-08-19T14:11:44.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-08-19T14:11:44.394 INFO:tasks.workunit.client.0.smithi102.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:44.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-08-19T14:11:44.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.44d2Lb7bVs_the_val 2024-08-19T14:11:44.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-08-19T14:11:44.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.44d2Lb7bVs_the_val' 2024-08-19T14:11:44.396 INFO:tasks.workunit.client.0.smithi102.stdout:Value dumped in: /tmp/tmp.44d2Lb7bVs_the_val 2024-08-19T14:11:44.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.44d2Lb7bVs_the_val 2024-08-19T14:11:44.986 INFO:tasks.workunit.client.0.smithi102.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-08-19T14:11:45.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-08-19T14:11:45.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.44d2Lb7bVs_the_val 2024-08-19T14:11:45.446 INFO:tasks.workunit.client.0.smithi102.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-08-19T14:11:45.446 INFO:tasks.workunit.client.0.smithi102.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-08-19T14:11:45.446 INFO:tasks.workunit.client.0.smithi102.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-08-19T14:11:45.446 INFO:tasks.workunit.client.0.smithi102.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-08-19T14:11:45.446 INFO:tasks.workunit.client.0.smithi102.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-08-19T14:11:45.446 INFO:tasks.workunit.client.0.smithi102.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-08-19T14:11:45.447 INFO:tasks.workunit.client.0.smithi102.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-08-19T14:11:45.447 INFO:tasks.workunit.client.0.smithi102.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-08-19T14:11:45.447 INFO:tasks.workunit.client.0.smithi102.stdout:0000073 2024-08-19T14:11:45.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-08-19T14:11:45.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:46.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.44d2Lb7bVs_the_val 2024-08-19T14:11:47.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-08-19T14:11:47.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-08-19T14:11:47.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-08-19T14:11:47.535 INFO:tasks.workunit.client.0.smithi102.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2024-08-19T14:11:47.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2024-08-19T14:11:47.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-08-19T14:11:47.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-08-19T14:11:47.561 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.560+0000 7f603c67c980 1 bdev(0x561c7f56e000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-08-19T14:11:47.561 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.560+0000 7f603c67c980 0 bdev(0x561c7f56e000 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-08-19T14:11:47.561 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.560+0000 7f603c67c980 1 bdev(0x561c7f56e000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-08-19T14:11:47.561 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.560+0000 7f603c67c980 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-08-19T14:11:47.561 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.560+0000 7f603c67c980 1 bdev(0x561c7f56e700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-08-19T14:11:47.561 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.560+0000 7f603c67c980 0 bdev(0x561c7f56e700 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-08-19T14:11:47.561 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.560+0000 7f603c67c980 1 bdev(0x561c7f56e700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-08-19T14:11:47.562 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.560+0000 7f603c67c980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-08-19T14:11:47.562 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.560+0000 7f603c67c980 1 bluefs mount 2024-08-19T14:11:47.566 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.566+0000 7f603c67c980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-08-19T14:11:47.575 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.575+0000 7f603c67c980 1 bluefs mount shared_bdev_used = 0 2024-08-19T14:11:47.576 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.575+0000 7f603c67c980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-08-19T14:11:47.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.577+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-08-19T14:11:47.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.577+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-08-19T14:11:47.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.577+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-08-19T14:11:47.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.577+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-08-19T14:11:47.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.578+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-08-19T14:11:47.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.578+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-08-19T14:11:47.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.578+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-08-19T14:11:47.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.578+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-08-19T14:11:47.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.578+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-08-19T14:11:47.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.578+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-08-19T14:11:47.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.578+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-08-19T14:11:47.580 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.578+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-08-19T14:11:47.585 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.583+0000 7f603c67c980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-08-19T14:11:47.585 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.583+0000 7f603c67c980 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2024-08-19T14:11:47.585 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.583+0000 7f603c67c980 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2024-08-19T14:11:47.585 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.583+0000 7f603c67c980 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-08-19T14:11:47.585 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.583+0000 7f603c67c980 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2024-08-19T14:11:47.585 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.584+0000 7f603c67c980 1 freelist init 2024-08-19T14:11:47.585 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.584+0000 7f603c67c980 1 freelist _read_cfg 2024-08-19T14:11:47.586 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.584+0000 7f603c67c980 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 16 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 5.72206e-07 2024-08-19T14:11:47.586 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.584+0000 7f603c67c980 1 bluefs umount 2024-08-19T14:11:47.586 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.584+0000 7f603c67c980 1 bdev(0x561c7f56e700 td/osd-mapper/1/block) close 2024-08-19T14:11:47.822 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.822+0000 7f603c67c980 1 bdev(0x561c7f56e700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-08-19T14:11:47.823 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.822+0000 7f603c67c980 0 bdev(0x561c7f56e700 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-08-19T14:11:47.823 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.822+0000 7f603c67c980 1 bdev(0x561c7f56e700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-08-19T14:11:47.823 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.822+0000 7f603c67c980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-08-19T14:11:47.823 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.822+0000 7f603c67c980 1 bluefs mount 2024-08-19T14:11:47.833 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.832+0000 7f603c67c980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-08-19T14:11:47.841 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.841+0000 7f603c67c980 1 bluefs mount shared_bdev_used = 9568256 2024-08-19T14:11:47.841 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.841+0000 7f603c67c980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-08-19T14:11:47.844 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.843+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-08-19T14:11:47.844 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.843+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-08-19T14:11:47.844 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.843+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-08-19T14:11:47.845 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.843+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-08-19T14:11:47.845 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.843+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-08-19T14:11:47.845 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.843+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-08-19T14:11:47.845 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.843+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-08-19T14:11:47.845 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.843+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-08-19T14:11:47.845 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.843+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-08-19T14:11:47.845 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.843+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-08-19T14:11:47.845 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.843+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-08-19T14:11:47.845 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:47.844+0000 7f603c67c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-08-19T14:11:48.097 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:48.096+0000 7f603c67c980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-08-19T14:11:48.098 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-08-19T14:11:48.099 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 0b 05 58 02 00 00 dc 43 96 5e 69 2e 49 2a b0 25 |..X....C.^i.I*.%| 2024-08-19T14:11:48.099 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 1c 03 b4 66 66 70 01 00 00 00 1c 00 00 00 00 00 |...ffp..........| 2024-08-19T14:11:48.099 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-19T14:11:48.099 INFO:tasks.workunit.client.0.smithi102.stdout:* 2024-08-19T14:11:48.099 INFO:tasks.workunit.client.0.smithi102.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-08-19T14:11:48.099 INFO:tasks.workunit.client.0.smithi102.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-08-19T14:11:48.099 INFO:tasks.workunit.client.0.smithi102.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-08-19T14:11:48.099 INFO:tasks.workunit.client.0.smithi102.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-08-19T14:11:48.099 INFO:tasks.workunit.client.0.smithi102.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-08-19T14:11:48.099 INFO:tasks.workunit.client.0.smithi102.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-08-19T14:11:48.099 INFO:tasks.workunit.client.0.smithi102.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-08-19T14:11:48.099 INFO:tasks.workunit.client.0.smithi102.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-08-19T14:11:48.099 INFO:tasks.workunit.client.0.smithi102.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-08-19T14:11:48.100 INFO:tasks.workunit.client.0.smithi102.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-08-19T14:11:48.100 INFO:tasks.workunit.client.0.smithi102.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-08-19T14:11:48.100 INFO:tasks.workunit.client.0.smithi102.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-08-19T14:11:48.100 INFO:tasks.workunit.client.0.smithi102.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-08-19T14:11:48.100 INFO:tasks.workunit.client.0.smithi102.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-08-19T14:11:48.100 INFO:tasks.workunit.client.0.smithi102.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-08-19T14:11:48.100 INFO:tasks.workunit.client.0.smithi102.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-08-19T14:11:48.100 INFO:tasks.workunit.client.0.smithi102.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-08-19T14:11:48.100 INFO:tasks.workunit.client.0.smithi102.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-08-19T14:11:48.100 INFO:tasks.workunit.client.0.smithi102.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-08-19T14:11:48.100 INFO:tasks.workunit.client.0.smithi102.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-08-19T14:11:48.100 INFO:tasks.workunit.client.0.smithi102.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-08-19T14:11:48.100 INFO:tasks.workunit.client.0.smithi102.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-08-19T14:11:48.100 INFO:tasks.workunit.client.0.smithi102.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-08-19T14:11:48.101 INFO:tasks.workunit.client.0.smithi102.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-08-19T14:11:48.101 INFO:tasks.workunit.client.0.smithi102.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-08-19T14:11:48.101 INFO:tasks.workunit.client.0.smithi102.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-08-19T14:11:48.101 INFO:tasks.workunit.client.0.smithi102.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-08-19T14:11:48.101 INFO:tasks.workunit.client.0.smithi102.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-08-19T14:11:48.101 INFO:tasks.workunit.client.0.smithi102.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-08-19T14:11:48.101 INFO:tasks.workunit.client.0.smithi102.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-08-19T14:11:48.101 INFO:tasks.workunit.client.0.smithi102.stdout:00000220 72 65 1c 00 00 00 0a 00 00 00 4f b2 3c de f6 a0 |re........O.<...| 2024-08-19T14:11:48.101 INFO:tasks.workunit.client.0.smithi102.stdout:00000230 4d 54 81 e5 e2 b0 09 ad 78 d2 00 00 00 00 00 00 |MT......x.......| 2024-08-19T14:11:48.101 INFO:tasks.workunit.client.0.smithi102.stdout:00000240 00 00 1c 00 00 00 61 52 c3 66 d7 f5 88 16 01 00 |......aR.f......| 2024-08-19T14:11:48.101 INFO:tasks.workunit.client.0.smithi102.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-08-19T14:11:48.102 INFO:tasks.workunit.client.0.smithi102.stdout:0000025e 2024-08-19T14:11:48.102 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:48.102 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-08-19T14:11:48.102 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:48.102 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-08-19T14:11:48.102 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-08-19T14:11:48.102 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-08-19T14:11:48.102 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-08-19T14:11:48.102 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-08-19T14:11:48.103 INFO:tasks.workunit.client.0.smithi102.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-08-19T14:11:48.103 INFO:tasks.workunit.client.0.smithi102.stdout:00000047 2024-08-19T14:11:48.103 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:48.103 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:48.103 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-08-19T14:11:48.103 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-08-19T14:11:48.103 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-08-19T14:11:48.103 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-08-19T14:11:48.103 INFO:tasks.workunit.client.0.smithi102.stdout:0000003f 2024-08-19T14:11:48.103 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:48.103 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-08-19T14:11:48.103 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-08-19T14:11:48.103 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-08-19T14:11:48.104 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-08-19T14:11:48.104 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-08-19T14:11:48.104 INFO:tasks.workunit.client.0.smithi102.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-08-19T14:11:48.104 INFO:tasks.workunit.client.0.smithi102.stdout:00000047 2024-08-19T14:11:48.104 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:48.104 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-08-19T14:11:48.104 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:48.104 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:48.104 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:48.104 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:48.104 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:48.104 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:48.104 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-08-19T14:11:48.105 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:48.105 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:48.105 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:48.105 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:48.105 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:48.105 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:48.105 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:48.105 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:48.105 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:48.105 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:48.105 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:48.105 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:48.105 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:48.105 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-08-19T14:11:48.106 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:48.106 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:48.106 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:48.106 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:48.106 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:48.106 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:48.106 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-08-19T14:11:48.106 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:48.106 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:48.106 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:48.106 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:48.106 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:48.107 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:48.107 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-08-19T14:11:48.107 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:48.107 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-08-19T14:11:48.107 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:48.107 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:48.097+0000 7f603c67c980 1 bluefs umount 2024-08-19T14:11:48.107 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:48.097+0000 7f603c67c980 1 bdev(0x561c7f56e700 td/osd-mapper/1/block) close 2024-08-19T14:11:48.337 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:48.337+0000 7f603c67c980 1 freelist shutdown 2024-08-19T14:11:48.337 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:48.337+0000 7f603c67c980 1 bdev(0x561c7f56e000 td/osd-mapper/1/block) close 2024-08-19T14:11:48.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-08-19T14:11:48.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-08-19T14:11:48.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-08-19T14:11:49.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:49.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-08-19T14:11:49.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-08-19T14:11:49.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-08-19T14:11:49.375 INFO:tasks.workunit.client.0.smithi102.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:49.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-08-19T14:11:49.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.njOU7IEfGY_the_val 2024-08-19T14:11:49.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-08-19T14:11:49.377 INFO:tasks.workunit.client.0.smithi102.stdout:Value dumped in: /tmp/tmp.njOU7IEfGY_the_val 2024-08-19T14:11:49.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.njOU7IEfGY_the_val' 2024-08-19T14:11:49.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.njOU7IEfGY_the_val 2024-08-19T14:11:49.908 INFO:tasks.workunit.client.0.smithi102.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-08-19T14:11:50.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-08-19T14:11:50.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.njOU7IEfGY_the_val 2024-08-19T14:11:50.174 INFO:tasks.workunit.client.0.smithi102.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-08-19T14:11:50.174 INFO:tasks.workunit.client.0.smithi102.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-08-19T14:11:50.174 INFO:tasks.workunit.client.0.smithi102.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-08-19T14:11:50.174 INFO:tasks.workunit.client.0.smithi102.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-08-19T14:11:50.174 INFO:tasks.workunit.client.0.smithi102.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-08-19T14:11:50.174 INFO:tasks.workunit.client.0.smithi102.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-08-19T14:11:50.174 INFO:tasks.workunit.client.0.smithi102.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-08-19T14:11:50.174 INFO:tasks.workunit.client.0.smithi102.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-08-19T14:11:50.174 INFO:tasks.workunit.client.0.smithi102.stdout:0000073 2024-08-19T14:11:50.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-08-19T14:11:50.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:51.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.njOU7IEfGY_the_val 2024-08-19T14:11:52.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-08-19T14:11:52.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-08-19T14:11:52.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2024-08-19T14:11:52.267 INFO:tasks.workunit.client.0.smithi102.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2024-08-19T14:11:52.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2024-08-19T14:11:52.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-08-19T14:11:52.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-08-19T14:11:52.292 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.291+0000 7f95485d5980 1 bdev(0x5619c01da000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-08-19T14:11:52.292 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.291+0000 7f95485d5980 0 bdev(0x5619c01da000 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-08-19T14:11:52.293 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.292+0000 7f95485d5980 1 bdev(0x5619c01da000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-08-19T14:11:52.293 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.292+0000 7f95485d5980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-08-19T14:11:52.293 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.292+0000 7f95485d5980 1 bdev(0x5619c01da700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-08-19T14:11:52.293 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.292+0000 7f95485d5980 0 bdev(0x5619c01da700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-08-19T14:11:52.293 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.292+0000 7f95485d5980 1 bdev(0x5619c01da700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-08-19T14:11:52.293 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.292+0000 7f95485d5980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-08-19T14:11:52.293 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.292+0000 7f95485d5980 1 bluefs mount 2024-08-19T14:11:52.303 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.303+0000 7f95485d5980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-08-19T14:11:52.313 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.312+0000 7f95485d5980 1 bluefs mount shared_bdev_used = 0 2024-08-19T14:11:52.313 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.312+0000 7f95485d5980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-08-19T14:11:52.316 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.314+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-08-19T14:11:52.316 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.314+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-08-19T14:11:52.316 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.314+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-08-19T14:11:52.316 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.315+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-08-19T14:11:52.316 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.315+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-08-19T14:11:52.316 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.315+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-08-19T14:11:52.316 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.315+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-08-19T14:11:52.316 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.315+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-08-19T14:11:52.316 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.315+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-08-19T14:11:52.317 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.315+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-08-19T14:11:52.317 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.315+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-08-19T14:11:52.317 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.315+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-08-19T14:11:52.321 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.320+0000 7f95485d5980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-08-19T14:11:52.321 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.320+0000 7f95485d5980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-08-19T14:11:52.321 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.320+0000 7f95485d5980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-08-19T14:11:52.321 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.320+0000 7f95485d5980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-08-19T14:11:52.321 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.320+0000 7f95485d5980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-08-19T14:11:52.321 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.321+0000 7f95485d5980 1 freelist init 2024-08-19T14:11:52.322 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.321+0000 7f95485d5980 1 freelist _read_cfg 2024-08-19T14:11:52.322 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.321+0000 7f95485d5980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 14 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 4.95912e-07 2024-08-19T14:11:52.322 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.321+0000 7f95485d5980 1 bluefs umount 2024-08-19T14:11:52.322 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.321+0000 7f95485d5980 1 bdev(0x5619c01da700 td/osd-mapper/2/block) close 2024-08-19T14:11:52.554 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.554+0000 7f95485d5980 1 bdev(0x5619c01da700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-08-19T14:11:52.555 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.554+0000 7f95485d5980 0 bdev(0x5619c01da700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-08-19T14:11:52.555 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.554+0000 7f95485d5980 1 bdev(0x5619c01da700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-08-19T14:11:52.555 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.554+0000 7f95485d5980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-08-19T14:11:52.555 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.554+0000 7f95485d5980 1 bluefs mount 2024-08-19T14:11:52.561 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.561+0000 7f95485d5980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-08-19T14:11:52.575 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.575+0000 7f95485d5980 1 bluefs mount shared_bdev_used = 9568256 2024-08-19T14:11:52.575 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.575+0000 7f95485d5980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-08-19T14:11:52.578 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.577+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-08-19T14:11:52.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.577+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-08-19T14:11:52.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.577+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-08-19T14:11:52.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.577+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-08-19T14:11:52.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.577+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-08-19T14:11:52.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.577+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-08-19T14:11:52.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.577+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-08-19T14:11:52.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.577+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-08-19T14:11:52.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.578+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-08-19T14:11:52.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.578+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-08-19T14:11:52.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.578+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-08-19T14:11:52.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.578+0000 7f95485d5980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-08-19T14:11:52.923 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.923+0000 7f95485d5980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-08-19T14:11:52.925 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-08-19T14:11:52.925 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 0b 05 58 02 00 00 dc 43 96 5e 69 2e 49 2a b0 25 |..X....C.^i.I*.%| 2024-08-19T14:11:52.926 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 1c 03 b4 66 66 70 02 00 00 00 1c 00 00 00 00 00 |...ffp..........| 2024-08-19T14:11:52.926 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-19T14:11:52.926 INFO:tasks.workunit.client.0.smithi102.stdout:* 2024-08-19T14:11:52.926 INFO:tasks.workunit.client.0.smithi102.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-08-19T14:11:52.926 INFO:tasks.workunit.client.0.smithi102.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-08-19T14:11:52.926 INFO:tasks.workunit.client.0.smithi102.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-08-19T14:11:52.926 INFO:tasks.workunit.client.0.smithi102.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-08-19T14:11:52.926 INFO:tasks.workunit.client.0.smithi102.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-08-19T14:11:52.926 INFO:tasks.workunit.client.0.smithi102.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-08-19T14:11:52.926 INFO:tasks.workunit.client.0.smithi102.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-08-19T14:11:52.926 INFO:tasks.workunit.client.0.smithi102.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-08-19T14:11:52.926 INFO:tasks.workunit.client.0.smithi102.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-08-19T14:11:52.927 INFO:tasks.workunit.client.0.smithi102.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-08-19T14:11:52.927 INFO:tasks.workunit.client.0.smithi102.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-08-19T14:11:52.927 INFO:tasks.workunit.client.0.smithi102.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-08-19T14:11:52.927 INFO:tasks.workunit.client.0.smithi102.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-08-19T14:11:52.927 INFO:tasks.workunit.client.0.smithi102.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-08-19T14:11:52.927 INFO:tasks.workunit.client.0.smithi102.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-08-19T14:11:52.927 INFO:tasks.workunit.client.0.smithi102.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-08-19T14:11:52.927 INFO:tasks.workunit.client.0.smithi102.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-08-19T14:11:52.927 INFO:tasks.workunit.client.0.smithi102.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-08-19T14:11:52.927 INFO:tasks.workunit.client.0.smithi102.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-08-19T14:11:52.927 INFO:tasks.workunit.client.0.smithi102.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-08-19T14:11:52.927 INFO:tasks.workunit.client.0.smithi102.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-08-19T14:11:52.927 INFO:tasks.workunit.client.0.smithi102.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-08-19T14:11:52.928 INFO:tasks.workunit.client.0.smithi102.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-08-19T14:11:52.928 INFO:tasks.workunit.client.0.smithi102.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-08-19T14:11:52.928 INFO:tasks.workunit.client.0.smithi102.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-08-19T14:11:52.928 INFO:tasks.workunit.client.0.smithi102.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-08-19T14:11:52.928 INFO:tasks.workunit.client.0.smithi102.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-08-19T14:11:52.928 INFO:tasks.workunit.client.0.smithi102.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-08-19T14:11:52.928 INFO:tasks.workunit.client.0.smithi102.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-08-19T14:11:52.928 INFO:tasks.workunit.client.0.smithi102.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-08-19T14:11:52.928 INFO:tasks.workunit.client.0.smithi102.stdout:00000220 72 65 1c 00 00 00 0f 00 00 00 22 8d 2e 03 bf d5 |re........".....| 2024-08-19T14:11:52.928 INFO:tasks.workunit.client.0.smithi102.stdout:00000230 46 c4 84 e7 0c ea 44 48 81 81 00 00 00 00 00 00 |F.....DH........| 2024-08-19T14:11:52.928 INFO:tasks.workunit.client.0.smithi102.stdout:00000240 00 00 1c 00 00 00 83 52 c3 66 bf 9d c2 09 01 00 |.......R.f......| 2024-08-19T14:11:52.928 INFO:tasks.workunit.client.0.smithi102.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-08-19T14:11:52.928 INFO:tasks.workunit.client.0.smithi102.stdout:0000025e 2024-08-19T14:11:52.928 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:52.929 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-08-19T14:11:52.929 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:52.929 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-08-19T14:11:52.929 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-08-19T14:11:52.929 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-08-19T14:11:52.929 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-08-19T14:11:52.929 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-08-19T14:11:52.929 INFO:tasks.workunit.client.0.smithi102.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-08-19T14:11:52.929 INFO:tasks.workunit.client.0.smithi102.stdout:00000047 2024-08-19T14:11:52.929 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:52.929 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:52.929 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-08-19T14:11:52.930 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-08-19T14:11:52.930 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-08-19T14:11:52.930 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-08-19T14:11:52.930 INFO:tasks.workunit.client.0.smithi102.stdout:0000003f 2024-08-19T14:11:52.930 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:52.930 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-08-19T14:11:52.930 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-08-19T14:11:52.930 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-08-19T14:11:52.930 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-08-19T14:11:52.930 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-08-19T14:11:52.930 INFO:tasks.workunit.client.0.smithi102.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-08-19T14:11:52.930 INFO:tasks.workunit.client.0.smithi102.stdout:00000047 2024-08-19T14:11:52.930 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:52.931 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-08-19T14:11:52.931 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:52.931 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:52.931 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:52.931 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:52.931 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:52.931 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:52.931 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-08-19T14:11:52.931 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:52.931 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:52.931 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:52.931 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:52.931 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:52.931 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:52.932 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:52.932 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:52.932 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:52.932 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:52.932 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:52.932 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:52.932 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:52.932 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-08-19T14:11:52.932 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:52.932 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:52.932 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:52.932 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:52.932 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:52.933 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:52.933 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-08-19T14:11:52.933 INFO:tasks.workunit.client.0.smithi102.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-08-19T14:11:52.933 INFO:tasks.workunit.client.0.smithi102.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-08-19T14:11:52.933 INFO:tasks.workunit.client.0.smithi102.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-08-19T14:11:52.933 INFO:tasks.workunit.client.0.smithi102.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-08-19T14:11:52.933 INFO:tasks.workunit.client.0.smithi102.stdout:0000003b 2024-08-19T14:11:52.933 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:52.933 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-08-19T14:11:52.933 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:52.933 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-08-19T14:11:52.933 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:11:52.934 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.924+0000 7f95485d5980 1 bluefs umount 2024-08-19T14:11:52.934 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:52.924+0000 7f95485d5980 1 bdev(0x5619c01da700 td/osd-mapper/2/block) close 2024-08-19T14:11:53.146 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:53.146+0000 7f95485d5980 1 freelist shutdown 2024-08-19T14:11:53.146 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:53.146+0000 7f95485d5980 1 bdev(0x5619c01da000 td/osd-mapper/2/block) close 2024-08-19T14:11:53.308 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-08-19T14:11:53.308 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-08-19T14:11:53.308 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-08-19T14:11:54.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:54.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-08-19T14:11:54.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-08-19T14:11:54.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-08-19T14:11:54.105 INFO:tasks.workunit.client.0.smithi102.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:54.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-08-19T14:11:54.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.2YfEuwGPoo_the_val 2024-08-19T14:11:54.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-08-19T14:11:54.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.2YfEuwGPoo_the_val' 2024-08-19T14:11:54.108 INFO:tasks.workunit.client.0.smithi102.stdout:Value dumped in: /tmp/tmp.2YfEuwGPoo_the_val 2024-08-19T14:11:54.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.2YfEuwGPoo_the_val 2024-08-19T14:11:54.617 INFO:tasks.workunit.client.0.smithi102.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-08-19T14:11:54.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-08-19T14:11:54.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.2YfEuwGPoo_the_val 2024-08-19T14:11:54.903 INFO:tasks.workunit.client.0.smithi102.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-08-19T14:11:54.903 INFO:tasks.workunit.client.0.smithi102.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-08-19T14:11:54.903 INFO:tasks.workunit.client.0.smithi102.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-08-19T14:11:54.903 INFO:tasks.workunit.client.0.smithi102.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-08-19T14:11:54.903 INFO:tasks.workunit.client.0.smithi102.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-08-19T14:11:54.903 INFO:tasks.workunit.client.0.smithi102.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-08-19T14:11:54.903 INFO:tasks.workunit.client.0.smithi102.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-08-19T14:11:54.903 INFO:tasks.workunit.client.0.smithi102.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-08-19T14:11:54.903 INFO:tasks.workunit.client.0.smithi102.stdout:0000073 2024-08-19T14:11:54.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-08-19T14:11:54.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:11:55.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.2YfEuwGPoo_the_val 2024-08-19T14:11:57.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-08-19T14:11:57.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 ' 2024-08-19T14:11:57.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 2024-08-19T14:11:57.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-08-19T14:11:57.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-08-19T14:11:57.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/' 2024-08-19T14:11:57.245 INFO:tasks.workunit.client.0.smithi102.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ 2024-08-19T14:11:57.246 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2024-08-19T14:11:57.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2024-08-19T14:11:57.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-08-19T14:11:57.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=dc43965e-692e-492a-b025-1c03b4666670 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-08-19T14:11:57.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2024-08-19T14:11:57.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-08-19T14:11:57.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:11:57.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T14:11:57.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:11:57.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/0 2024-08-19T14:11:57.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=dc43965e-692e-492a-b025-1c03b4666670 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-08-19T14:11:57.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:11:57.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:11:57.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:11:57.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2024-08-19T14:11:57.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2024-08-19T14:11:57.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:11:57.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:11:57.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-08-19T14:11:57.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:11:57.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:11:57.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:11:57.252 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:11:57.252 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:11:57.252 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.15855 2024-08-19T14:11:57.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.15855/$cluster-$name.asok' 2024-08-19T14:11:57.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.15855/$cluster-$name.asok' 2024-08-19T14:11:57.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:11:57.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-08-19T14:11:57.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-08-19T14:11:57.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:11:57.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:11:57.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:11:57.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:11:57.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:11:57.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:11:57.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/0 2024-08-19T14:11:57.253 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T14:11:57.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T14:11:57.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=dc43965e-692e-492a-b025-1c03b4666670 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.15855/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:11:57.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/0/whoami 2024-08-19T14:11:57.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T14:11:57.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:11:57.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:11:57.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:11:57.291 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:57.290+0000 7fcddb00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:11:57.293 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:57.292+0000 7fcddb00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:11:57.294 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:57.294+0000 7fcddb00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:11:57.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T14:11:57.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:11:57.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:11:57.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:11:57.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:11:57.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:11:57.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:11:57.541 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:11:57.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:11:57.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:11:57.819 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:11:57.819+0000 7fcddb00f640 -1 Falling back to public interface 2024-08-19T14:11:57.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:11:58.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:11:58.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:11:58.862 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:11:58.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:11:58.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:11:58.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:11:59.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:12:00.012 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:00.011+0000 7fcddb00f640 -1 osd.0 28 log_to_monitors true 2024-08-19T14:12:00.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:12:00.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:12:00.149 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:12:00.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:12:00.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:12:00.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:12:00.436 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 31 up_thru 17 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/301196895,v1:127.0.0.1:6803/301196895] [v2:127.0.0.1:6804/301196895,v1:127.0.0.1:6805/301196895] exists,up 3cab53a5-c985-4dbb-bc71-967899357988 2024-08-19T14:12:00.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:12:00.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:12:00.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:12:00.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-08-19T14:12:00.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=dc43965e-692e-492a-b025-1c03b4666670 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-08-19T14:12:00.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2024-08-19T14:12:00.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-08-19T14:12:00.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:12:00.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:12:00.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:12:00.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/1 2024-08-19T14:12:00.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=dc43965e-692e-492a-b025-1c03b4666670 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-08-19T14:12:00.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:12:00.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:12:00.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:12:00.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2024-08-19T14:12:00.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2024-08-19T14:12:00.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:12:00.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:12:00.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-08-19T14:12:00.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:12:00.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:12:00.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:12:00.442 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:12:00.443 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:12:00.443 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.15855 2024-08-19T14:12:00.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.15855/$cluster-$name.asok' 2024-08-19T14:12:00.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.15855/$cluster-$name.asok' 2024-08-19T14:12:00.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:12:00.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-08-19T14:12:00.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-08-19T14:12:00.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:12:00.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:12:00.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:12:00.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:12:00.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:12:00.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:12:00.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/1 2024-08-19T14:12:00.444 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T14:12:00.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:12:00.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=dc43965e-692e-492a-b025-1c03b4666670 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.15855/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:12:00.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/1/whoami 2024-08-19T14:12:00.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:12:00.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:12:00.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:12:00.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:12:00.476 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:00.475+0000 7f6ab5da7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:00.546 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:00.545+0000 7f6ab5da7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:00.547 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:00.546+0000 7f6ab5da7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:00.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:12:00.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:12:00.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:12:00.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:12:00.835 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:12:00.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:12:00.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:12:00.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:12:00.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:12:00.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:12:01.063 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:01.062+0000 7f6ab5da7640 -1 Falling back to public interface 2024-08-19T14:12:01.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:12:02.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:12:02.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:12:02.172 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:12:02.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:12:02.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:12:02.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:12:02.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:12:03.321 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:03.319+0000 7f6ab5da7640 -1 osd.1 28 log_to_monitors true 2024-08-19T14:12:03.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:12:03.453 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:12:03.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:12:03.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:12:03.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:12:03.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:12:03.859 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 34 up_thru 17 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/3373705884,v1:127.0.0.1:6811/3373705884] [v2:127.0.0.1:6812/3373705884,v1:127.0.0.1:6813/3373705884] exists,up 4fb23cde-f6a0-4d54-81e5-e2b009ad78d2 2024-08-19T14:12:03.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:12:03.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:12:03.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:12:03.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-08-19T14:12:03.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=dc43965e-692e-492a-b025-1c03b4666670 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-08-19T14:12:03.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2024-08-19T14:12:03.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-08-19T14:12:03.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:12:03.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-19T14:12:03.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:12:03.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/2 2024-08-19T14:12:03.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=dc43965e-692e-492a-b025-1c03b4666670 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-08-19T14:12:03.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:12:03.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:12:03.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:12:03.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2024-08-19T14:12:03.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2024-08-19T14:12:03.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:12:03.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:12:03.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-08-19T14:12:03.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:12:03.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:12:03.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:12:03.865 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:12:03.865 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:12:03.865 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.15855 2024-08-19T14:12:03.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.15855/$cluster-$name.asok' 2024-08-19T14:12:03.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.15855/$cluster-$name.asok' 2024-08-19T14:12:03.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:12:03.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-08-19T14:12:03.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-08-19T14:12:03.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:12:03.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:12:03.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:12:03.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:12:03.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:12:03.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:12:03.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/2 2024-08-19T14:12:03.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-19T14:12:03.866 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T14:12:03.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=dc43965e-692e-492a-b025-1c03b4666670 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.15855/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:12:03.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/2/whoami 2024-08-19T14:12:03.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-19T14:12:03.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:12:03.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:12:03.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:12:03.900 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:03.899+0000 7f61cc25d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:03.922 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:03.922+0000 7f61cc25d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:03.923 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:03.923+0000 7f61cc25d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:04.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-19T14:12:04.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:12:04.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T14:12:04.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:12:04.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:12:04.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:12:04.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:12:04.295 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:12:04.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:12:04.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:12:04.439 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:04.439+0000 7f61cc25d640 -1 Falling back to public interface 2024-08-19T14:12:04.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:12:05.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:12:05.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:12:05.605 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:12:05.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:12:05.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:12:05.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:12:05.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:12:06.692 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:06.692+0000 7f61cc25d640 -1 osd.2 28 log_to_monitors true 2024-08-19T14:12:06.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:12:06.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:12:06.917 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:12:06.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:12:06.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:12:06.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:12:07.353 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 38 up_thru 17 down_at 29 last_clean_interval [15,28) [v2:127.0.0.1:6818/4101839515,v1:127.0.0.1:6819/4101839515] [v2:127.0.0.1:6820/4101839515,v1:127.0.0.1:6821/4101839515] exists,up 228d2e03-bfd5-46c4-84e7-0cea44488181 2024-08-19T14:12:07.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:12:07.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:12:07.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:12:07.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2024-08-19T14:12:08.356 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2024-08-19T14:12:08.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2024-08-19T14:12:08.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-08-19T14:12:08.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2024-08-19T14:12:08.464 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T14:12:08.464 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "19.1.0-1382-gde09dd01", 2024-08-19T14:12:08.464 INFO:tasks.workunit.client.0.smithi102.stdout: "release": "squid", 2024-08-19T14:12:08.464 INFO:tasks.workunit.client.0.smithi102.stdout: "release_type": "rc" 2024-08-19T14:12:08.465 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:12:08.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-08-19T14:12:08.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2024-08-19T14:12:08.571 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T14:12:08.571 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "19.1.0-1382-gde09dd01", 2024-08-19T14:12:08.572 INFO:tasks.workunit.client.0.smithi102.stdout: "release": "squid", 2024-08-19T14:12:08.572 INFO:tasks.workunit.client.0.smithi102.stdout: "release_type": "rc" 2024-08-19T14:12:08.572 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:12:08.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-08-19T14:12:08.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2024-08-19T14:12:08.678 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T14:12:08.678 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "19.1.0-1382-gde09dd01", 2024-08-19T14:12:08.678 INFO:tasks.workunit.client.0.smithi102.stdout: "release": "squid", 2024-08-19T14:12:08.678 INFO:tasks.workunit.client.0.smithi102.stdout: "release_type": "rc" 2024-08-19T14:12:08.679 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:12:08.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-08-19T14:12:08.737 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T14:12:08.737 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "objxxx", 2024-08-19T14:12:08.737 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T14:12:08.737 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T14:12:08.737 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-08-19T14:12:08.737 INFO:tasks.workunit.client.0.smithi102.stdout: "snapshots": [ 2024-08-19T14:12:08.737 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T14:12:08.737 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-08-19T14:12:08.737 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "snap01" 2024-08-19T14:12:08.738 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T14:12:08.738 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T14:12:08.738 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-08-19T14:12:08.738 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "snap02" 2024-08-19T14:12:08.738 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T14:12:08.738 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T14:12:08.738 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 6, 2024-08-19T14:12:08.738 INFO:tasks.workunit.client.0.smithi102.stdout: "overlaps": [] 2024-08-19T14:12:08.738 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T14:12:08.738 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T14:12:08.738 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 3, 2024-08-19T14:12:08.738 INFO:tasks.workunit.client.0.smithi102.stdout: "snapshots": [ 2024-08-19T14:12:08.739 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T14:12:08.739 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 3, 2024-08-19T14:12:08.739 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "snap13" 2024-08-19T14:12:08.739 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T14:12:08.739 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T14:12:08.739 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 6, 2024-08-19T14:12:08.739 INFO:tasks.workunit.client.0.smithi102.stdout: "overlaps": [] 2024-08-19T14:12:08.739 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T14:12:08.739 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T14:12:08.739 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 5, 2024-08-19T14:12:08.739 INFO:tasks.workunit.client.0.smithi102.stdout: "snapshots": [ 2024-08-19T14:12:08.739 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T14:12:08.739 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 4, 2024-08-19T14:12:08.739 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "snap24" 2024-08-19T14:12:08.740 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T14:12:08.740 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T14:12:08.740 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 5, 2024-08-19T14:12:08.740 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "snap25" 2024-08-19T14:12:08.740 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T14:12:08.740 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T14:12:08.740 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 5, 2024-08-19T14:12:08.740 INFO:tasks.workunit.client.0.smithi102.stdout: "overlaps": [] 2024-08-19T14:12:08.740 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T14:12:08.740 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T14:12:08.740 INFO:tasks.workunit.client.0.smithi102.stdout: "id": "head", 2024-08-19T14:12:08.740 INFO:tasks.workunit.client.0.smithi102.stdout: "snapshots": [], 2024-08-19T14:12:08.740 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 6 2024-08-19T14:12:08.740 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T14:12:08.741 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T14:12:08.741 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:12:08.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2024-08-19T14:12:09.083 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is unset 2024-08-19T14:12:09.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2024-08-19T14:12:09.645 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is unset 2024-08-19T14:12:09.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-08-19T14:12:09.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2024-08-19T14:12:09.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2024-08-19T14:12:09.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2024-08-19T14:12:10.181 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T14:12:10.181 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-08-19T14:12:08.140720+0000 26'7 40:64 [1,2,0] 1 [1,2,0] 1 26'7 2024-08-19T14:11:28.139046+0000 26'7 2024-08-19T14:11:28.139046+0000 0 1 periodic scrub scheduled @ 2024-08-20T14:11:28.139046+0000 4 0 2024-08-19T14:12:10.181 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-08-19T14:12:04.618123+0000 0'0 36:27 [0,1] 0 [0,1] 0 0'0 2024-08-19T14:11:16.685013+0000 0'0 2024-08-19T14:11:16.685013+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:12:10.181 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-08-19T14:12:04.618132+0000 0'0 36:18 [0,1] 0 [0,1] 0 0'0 2024-08-19T14:11:16.685013+0000 0'0 2024-08-19T14:11:16.685013+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:11:16.685013+0000 0 0 2024-08-19T14:12:10.181 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:12:08.140744+0000 0'0 40:34 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-19T14:11:16.685013+0000 0'0 2024-08-19T14:11:16.685013+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:11:16.685013+0000 0 0 2024-08-19T14:12:10.181 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:12:10.181 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T14:12:10.182 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:12:10.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2024-08-19T14:12:12.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-08-19T14:12:12.293 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T14:12:12.293 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": true, 2024-08-19T14:12:12.293 INFO:tasks.workunit.client.0.smithi102.stdout: "must": true, 2024-08-19T14:12:12.293 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "0.000000" 2024-08-19T14:12:12.293 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:12:12.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2024-08-19T14:12:17.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2024-08-19T14:12:17.556 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T14:12:17.556 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-08-19T14:12:12.477219+0000 26'7 42:78 [1,2,0] 1 [1,2,0] 1 26'7 2024-08-19T14:12:12.477146+0000 26'7 2024-08-19T14:12:12.477146+0000 0 1 periodic scrub scheduled @ 2024-08-20T14:12:12.477146+0000 4 0 2024-08-19T14:12:17.556 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:12:07.705270+0000 0'0 42:44 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-19T14:11:16.685013+0000 0'0 2024-08-19T14:11:16.685013+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:11:16.685013+0000 0 0 2024-08-19T14:12:17.556 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:12:07.705271+0000 0'0 42:28 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-19T14:11:16.685013+0000 0'0 2024-08-19T14:11:16.685013+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:11:16.685013+0000 0 0 2024-08-19T14:12:17.556 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:12:08.140744+0000 0'0 42:38 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-19T14:11:16.685013+0000 0'0 2024-08-19T14:11:16.685013+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:11:16.685013+0000 0 0 2024-08-19T14:12:17.556 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:12:17.556 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T14:12:17.556 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:12:17.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-08-19T14:12:17.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-08-19T14:12:17.568 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:12:12.476+0000 7f6a9851d640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-08-19T14:12:17.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2024-08-19T14:12:17.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-08-19T14:12:17.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2024-08-19T14:12:17.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2024-08-19T14:12:17.573 INFO:tasks.workunit.client.0.smithi102.stdout:prev count: 1 2024-08-19T14:12:17.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2024-08-19T14:12:17.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-08-19T14:12:17.665 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T14:12:17.665 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": true, 2024-08-19T14:12:17.665 INFO:tasks.workunit.client.0.smithi102.stdout: "must": true, 2024-08-19T14:12:17.665 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "0.000000" 2024-08-19T14:12:17.665 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:12:17.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2024-08-19T14:12:22.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2024-08-19T14:12:22.936 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T14:12:22.936 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-08-19T14:12:18.431419+0000 26'7 42:87 [1,2,0] 1 [1,2,0] 1 26'7 2024-08-19T14:12:18.431325+0000 26'7 2024-08-19T14:12:18.431325+0000 0 1 periodic scrub scheduled @ 2024-08-20T14:12:18.431325+0000 4 0 2024-08-19T14:12:22.936 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:12:07.705270+0000 0'0 42:44 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-19T14:11:16.685013+0000 0'0 2024-08-19T14:11:16.685013+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:11:16.685013+0000 0 0 2024-08-19T14:12:22.936 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:12:07.705271+0000 0'0 42:28 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-19T14:11:16.685013+0000 0'0 2024-08-19T14:11:16.685013+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:11:16.685013+0000 0 0 2024-08-19T14:12:22.936 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:12:08.140744+0000 0'0 42:38 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-19T14:11:16.685013+0000 0'0 2024-08-19T14:11:16.685013+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:11:16.685013+0000 0 0 2024-08-19T14:12:22.936 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:12:22.937 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T14:12:22.937 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:12:22.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-08-19T14:12:22.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-08-19T14:12:22.948 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:12:12.476+0000 7f6a9851d640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-08-19T14:12:22.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-08-19T14:12:22.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2024-08-19T14:12:22.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2024-08-19T14:12:22.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-08-19T14:12:22.951 INFO:tasks.workunit.client.0.smithi102.stdout:current count: 1 2024-08-19T14:12:22.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2024-08-19T14:12:22.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2024-08-19T14:12:22.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-08-19T14:12:22.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:12:22.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:12:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:12:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:12:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:12:23.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:12:23.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-08-19T14:12:23.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-08-19T14:12:23.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-08-19T14:12:23.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2024-08-19T14:12:23.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-08-19T14:12:24.802 INFO:tasks.workunit.client.0.smithi102.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-08-19T14:12:24.802 INFO:tasks.workunit.client.0.smithi102.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-08-19T14:12:24.802 INFO:tasks.workunit.client.0.smithi102.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-08-19T14:12:24.802 INFO:tasks.workunit.client.0.smithi102.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-08-19T14:12:24.802 INFO:tasks.workunit.client.0.smithi102.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-08-19T14:12:24.803 INFO:tasks.workunit.client.0.smithi102.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-08-19T14:12:24.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-08-19T14:12:24.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-08-19T14:12:24.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-08-19T14:12:24.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2024-08-19T14:12:25.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2024-08-19T14:12:25.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2024-08-19T14:12:25.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2024-08-19T14:12:25.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2024-08-19T14:12:25.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-08-19T14:12:25.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:12:25.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-08-19T14:12:25.605 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:12:25.605 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:12:25.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:12:25.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:12:25.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:12:25.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:12:25.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:12:25.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:12:25.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:12:25.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:12:25.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:12:25.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:12:25.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:12:25.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:12:25.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:12:25.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:12:25.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:12:25.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:12:25.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-08-19T14:12:25.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:12:25.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:12:25.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.15855 2024-08-19T14:12:25.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.15855 2024-08-19T14:12:25.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:12:25.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:12:25.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-08-19T14:12:25.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-mapper 0 2024-08-19T14:12:25.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-08-19T14:12:25.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-08-19T14:12:25.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-08-19T14:12:25.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:12:25.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:12:25.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:12:25.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:12:25.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:12:25.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:12:25.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:12:25.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:12:25.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:12:25.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:12:25.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:12:25.742 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:12:25.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:12:25.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:12:25.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:12:25.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:12:25.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:12:25.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-08-19T14:12:25.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-08-19T14:12:25.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:12:25.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:12:25.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.15855 2024-08-19T14:12:25.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.15855 2024-08-19T14:12:25.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:12:25.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:12:25.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-08-19T14:12:25.751 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-19T14:12:25.751 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-19T14:12:25.819 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2024-08-19T14:12:25.819 DEBUG:teuthology.orchestra.run.smithi102:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=78215f5a1cf5eaf321d506a66539942e12851f34 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2024-08-19T14:12:25.884 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-19T14:12:25.889 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-19T14:12:25.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-08-19T14:12:25.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-08-19T14:12:25.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-08-19T14:12:25.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-08-19T14:12:25.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-08-19T14:12:25.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-08-19T14:12:25.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-08-19T14:12:25.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-08-19T14:12:25.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-recovery-scrub 2024-08-19T14:12:25.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2024-08-19T14:12:25.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2024-08-19T14:12:25.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2024-08-19T14:12:25.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2024-08-19T14:12:25.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2024-08-19T14:12:25.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2024-08-19T14:12:25.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none ' 2024-08-19T14:12:25.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-08-19T14:12:25.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-08-19T14:12:25.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-08-19T14:12:25.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2024-08-19T14:12:25.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-08-19T14:12:25.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local 'funcs=TEST_recovery_scrub_1 2024-08-19T14:12:25.895 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_recovery_scrub_2' 2024-08-19T14:12:25.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-08-19T14:12:25.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2024-08-19T14:12:25.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2024-08-19T14:12:25.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2024-08-19T14:12:25.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.32181 2024-08-19T14:12:25.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2024-08-19T14:12:25.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2024-08-19T14:12:25.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2024-08-19T14:12:25.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2024-08-19T14:12:25.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2024-08-19T14:12:25.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-08-19T14:12:25.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-08-19T14:12:25.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-08-19T14:12:25.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:12:25.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-08-19T14:12:25.897 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:12:25.897 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:12:25.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:12:25.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:12:25.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:12:25.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:12:25.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:12:25.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:12:25.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:12:25.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:12:25.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:12:25.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:12:25.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:12:25.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:12:25.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:12:25.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:12:25.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:12:25.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:12:25.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-08-19T14:12:25.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:12:25.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:12:25.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:12:25.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32181 2024-08-19T14:12:25.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:12:25.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:12:25.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-08-19T14:12:25.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T14:12:25.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:12:25.903 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:12:25.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.32181 2024-08-19T14:12:25.904 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T14:12:25.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-19T14:12:25.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-19T14:12:25.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T14:12:25.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-08-19T14:12:25.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T14:12:25.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-08-19T14:12:25.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T14:12:25.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T14:12:25.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T14:12:25.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-08-19T14:12:25.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T14:12:26.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T14:12:26.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:12:26.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:12:26.289 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:12:26.289 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:12:26.289 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:12:26.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:12:26.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T14:12:26.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T14:12:26.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T14:12:26.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:12:26.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:12:26.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T14:12:26.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T14:12:26.474 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:12:26.474 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:12:26.474 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:12:26.475 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:12:26.475 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:12:26.475 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:12:26.475 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32181/ceph-mon.a.asok 2024-08-19T14:12:26.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:12:26.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32181/ceph-mon.a.asok config get fsid 2024-08-19T14:12:26.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T14:12:26.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:12:26.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:12:26.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T14:12:26.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T14:12:26.562 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:12:26.562 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:12:26.562 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:12:26.562 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:12:26.562 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:12:26.562 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:12:26.562 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32181/ceph-mon.a.asok 2024-08-19T14:12:26.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:12:26.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32181/ceph-mon.a.asok config get mon_host 2024-08-19T14:12:26.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2024-08-19T14:12:26.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-08-19T14:12:26.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T14:12:26.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T14:12:26.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T14:12:26.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-08-19T14:12:26.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T14:12:27.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T14:12:27.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:12:27.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:12:27.450 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:12:27.450 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:12:27.450 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:12:27.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:12:27.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:12:27.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:12:27.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-08-19T14:12:27.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-08-19T14:12:27.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-08-19T14:12:27.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-08-19T14:12:27.487 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2024-08-19T14:12:27.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2024-08-19T14:12:27.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:12:27.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2024-08-19T14:12:27.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-08-19T14:12:27.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:12:27.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T14:12:27.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:12:27.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-08-19T14:12:27.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-08-19T14:12:27.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:12:27.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:12:27.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:12:27.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-08-19T14:12:27.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-08-19T14:12:27.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:12:27.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:12:27.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-08-19T14:12:27.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:12:27.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:12:27.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:12:27.489 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:12:27.490 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:12:27.490 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:12:27.490 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:12:27.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:12:27.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:12:27.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:12:27.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:12:27.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-08-19T14:12:27.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-08-19T14:12:27.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:12:27.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:12:27.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:12:27.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:12:27.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:12:27.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-08-19T14:12:27.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-08-19T14:12:27.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:12:27.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=912be436-6149-4a0a-b32a-51e69949b1c8 2024-08-19T14:12:27.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 912be436-6149-4a0a-b32a-51e69949b1c8' 2024-08-19T14:12:27.491 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 912be436-6149-4a0a-b32a-51e69949b1c8 2024-08-19T14:12:27.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:12:27.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLUsNm+yj9HRAAWx7NM2ak7lhUo+GAOpllFw== 2024-08-19T14:12:27.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLUsNm+yj9HRAAWx7NM2ak7lhUo+GAOpllFw=="}' 2024-08-19T14:12:27.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 912be436-6149-4a0a-b32a-51e69949b1c8 -i td/osd-recovery-scrub/0/new.json 2024-08-19T14:12:27.705 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:12:27.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-08-19T14:12:27.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQDLUsNm+yj9HRAAWx7NM2ak7lhUo+GAOpllFw== --osd-uuid 912be436-6149-4a0a-b32a-51e69949b1c8 2024-08-19T14:12:27.745 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:27.744+0000 7fcdc3c32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:27.756 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:27.756+0000 7fcdc3c32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:27.757 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:27.757+0000 7fcdc3c32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:27.758 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:27.757+0000 7fcdc3c32640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-08-19T14:12:27.758 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:27.757+0000 7fcdc3c32640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-08-19T14:12:27.758 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:27.757+0000 7fcdc3c32640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-08-19T14:12:27.848 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:27.848+0000 7fcdc3c32640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-08-19T14:12:33.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-08-19T14:12:33.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:12:33.188 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T14:12:33.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T14:12:33.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:12:33.540 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T14:12:33.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T14:12:33.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-08-19T14:12:33.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:12:33.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:12:33.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:12:33.567 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:33.567+0000 7f0c3e03f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:33.593 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:33.592+0000 7f0c3e03f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:33.594 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:33.593+0000 7f0c3e03f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:33.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T14:12:33.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:12:33.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:12:33.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:12:33.807 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:12:33.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:12:33.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:12:33.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:12:33.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:12:33.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:12:34.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:12:34.110 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:34.110+0000 7f0c3e03f640 -1 Falling back to public interface 2024-08-19T14:12:35.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:12:35.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:12:35.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:12:35.082 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:12:35.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:12:35.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:12:35.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:12:36.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:12:36.351 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:12:36.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:12:36.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:12:36.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:12:36.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:12:36.465 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:36.465+0000 7f0c3e03f640 -1 osd.0 0 log_to_monitors true 2024-08-19T14:12:36.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:12:37.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:12:37.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:12:37.641 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:12:37.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:12:37.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:12:37.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:12:37.947 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3459818255,v1:127.0.0.1:6803/3459818255] [v2:127.0.0.1:6804/3459818255,v1:127.0.0.1:6805/3459818255] exists,up 912be436-6149-4a0a-b32a-51e69949b1c8 2024-08-19T14:12:37.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:12:37.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:12:37.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:12:37.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:12:37.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2024-08-19T14:12:37.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-08-19T14:12:37.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:12:37.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T14:12:37.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:12:37.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-08-19T14:12:37.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-08-19T14:12:37.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:12:37.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:12:37.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:12:37.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-08-19T14:12:37.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-08-19T14:12:37.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:12:37.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:12:37.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-08-19T14:12:37.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:12:37.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:12:37.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:12:37.953 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:12:37.953 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:12:37.953 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:12:37.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:12:37.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:12:37.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:12:37.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:12:37.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:12:37.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-08-19T14:12:37.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-08-19T14:12:37.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:12:37.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:12:37.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:12:37.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:12:37.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:12:37.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-08-19T14:12:37.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-08-19T14:12:37.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:12:37.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d92da0b4-ac70-4401-a32a-4e7671a018d6 2024-08-19T14:12:37.955 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 d92da0b4-ac70-4401-a32a-4e7671a018d6 2024-08-19T14:12:37.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d92da0b4-ac70-4401-a32a-4e7671a018d6' 2024-08-19T14:12:37.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:12:37.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDVUsNmW0itORAAFOtVhP3cCrHC736AHMZbtQ== 2024-08-19T14:12:37.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDVUsNmW0itORAAFOtVhP3cCrHC736AHMZbtQ=="}' 2024-08-19T14:12:37.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d92da0b4-ac70-4401-a32a-4e7671a018d6 -i td/osd-recovery-scrub/1/new.json 2024-08-19T14:12:38.277 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:12:38.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-08-19T14:12:38.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQDVUsNmW0itORAAFOtVhP3cCrHC736AHMZbtQ== --osd-uuid d92da0b4-ac70-4401-a32a-4e7671a018d6 2024-08-19T14:12:38.317 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:38.317+0000 7f6341824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:38.362 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:38.361+0000 7f6341824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:38.363 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:38.362+0000 7f6341824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:38.363 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:38.363+0000 7f6341824640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-08-19T14:12:38.363 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:38.363+0000 7f6341824640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-08-19T14:12:38.363 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:38.363+0000 7f6341824640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-08-19T14:12:38.439 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:38.438+0000 7f6341824640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-08-19T14:12:44.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-08-19T14:12:44.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:12:44.262 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T14:12:44.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T14:12:44.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:12:44.606 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T14:12:44.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T14:12:44.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-08-19T14:12:44.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:12:44.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:12:44.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:12:44.633 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:44.632+0000 7f7c1cbd4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:44.635 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:44.634+0000 7f7c1cbd4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:44.636 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:44.635+0000 7f7c1cbd4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:44.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T14:12:44.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:12:44.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:12:44.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:12:44.873 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:12:44.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:12:44.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:12:44.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:12:44.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:12:44.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:12:45.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:12:45.162 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:45.162+0000 7f7c1cbd4640 -1 Falling back to public interface 2024-08-19T14:12:46.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:12:46.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:12:46.137 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:12:46.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:12:46.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:12:46.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:12:46.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:12:47.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:12:47.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:12:47.410 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:12:47.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:12:47.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:12:47.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:12:47.535 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:47.534+0000 7f7c1cbd4640 -1 osd.1 0 log_to_monitors true 2024-08-19T14:12:47.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:12:48.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:12:48.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:12:48.736 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:12:48.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:12:48.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:12:48.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:12:49.004 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4005434277,v1:127.0.0.1:6811/4005434277] [v2:127.0.0.1:6812/4005434277,v1:127.0.0.1:6813/4005434277] exists,up d92da0b4-ac70-4401-a32a-4e7671a018d6 2024-08-19T14:12:49.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:12:49.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:12:49.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:12:49.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:12:49.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2024-08-19T14:12:49.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-08-19T14:12:49.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:12:49.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T14:12:49.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:12:49.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-08-19T14:12:49.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-08-19T14:12:49.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:12:49.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:12:49.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:12:49.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-08-19T14:12:49.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-08-19T14:12:49.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:12:49.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:12:49.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-08-19T14:12:49.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:12:49.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:12:49.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:12:49.009 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:12:49.009 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:12:49.009 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:12:49.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:12:49.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:12:49.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:12:49.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:12:49.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:12:49.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-08-19T14:12:49.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-08-19T14:12:49.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:12:49.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:12:49.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:12:49.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:12:49.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:12:49.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-08-19T14:12:49.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-08-19T14:12:49.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:12:49.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=08fcf60b-3d3b-4caf-a1da-1f0139b0061f 2024-08-19T14:12:49.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 08fcf60b-3d3b-4caf-a1da-1f0139b0061f' 2024-08-19T14:12:49.011 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 08fcf60b-3d3b-4caf-a1da-1f0139b0061f 2024-08-19T14:12:49.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:12:49.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDhUsNm42eBARAAfmM3GVAFR7NOYFP3IpfZ+A== 2024-08-19T14:12:49.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDhUsNm42eBARAAfmM3GVAFR7NOYFP3IpfZ+A=="}' 2024-08-19T14:12:49.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 08fcf60b-3d3b-4caf-a1da-1f0139b0061f -i td/osd-recovery-scrub/2/new.json 2024-08-19T14:12:49.380 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:12:49.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-08-19T14:12:49.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQDhUsNm42eBARAAfmM3GVAFR7NOYFP3IpfZ+A== --osd-uuid 08fcf60b-3d3b-4caf-a1da-1f0139b0061f 2024-08-19T14:12:49.421 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:49.420+0000 7fc440a2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:49.457 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:49.456+0000 7fc440a2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:49.458 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:49.457+0000 7fc440a2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:49.458 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:49.458+0000 7fc440a2e640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-08-19T14:12:49.458 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:49.458+0000 7fc440a2e640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-08-19T14:12:49.458 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:49.458+0000 7fc440a2e640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-08-19T14:12:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:49.531+0000 7fc440a2e640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-08-19T14:12:55.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-08-19T14:12:55.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:12:55.405 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T14:12:55.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T14:12:55.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:12:55.753 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T14:12:55.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T14:12:55.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-08-19T14:12:55.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:12:55.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:12:55.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:12:55.780 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:55.779+0000 7f4822188640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:55.781 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:55.781+0000 7f4822188640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:55.783 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:55.782+0000 7f4822188640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:12:56.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T14:12:56.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:12:56.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T14:12:56.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:12:56.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:12:56.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:12:56.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:12:56.021 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:12:56.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:12:56.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:12:56.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:12:56.307 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:56.307+0000 7f4822188640 -1 Falling back to public interface 2024-08-19T14:12:57.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:12:57.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:12:57.285 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:12:57.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:12:57.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:12:57.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:12:57.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:12:58.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:12:58.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:12:58.557 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:12:58.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:12:58.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:12:58.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:12:58.639 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:12:58.638+0000 7f4822188640 -1 osd.2 0 log_to_monitors true 2024-08-19T14:12:58.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:12:59.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:12:59.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:12:59.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:12:59.839 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:12:59.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:12:59.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:13:00.122 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/696353818,v1:127.0.0.1:6819/696353818] [v2:127.0.0.1:6820/696353818,v1:127.0.0.1:6821/696353818] exists,up 08fcf60b-3d3b-4caf-a1da-1f0139b0061f 2024-08-19T14:13:00.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:13:00.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:13:00.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:13:00.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:13:00.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2024-08-19T14:13:00.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-08-19T14:13:00.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:13:00.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-08-19T14:13:00.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:13:00.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-08-19T14:13:00.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-08-19T14:13:00.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:13:00.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:13:00.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:13:00.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-08-19T14:13:00.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-08-19T14:13:00.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:13:00.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:13:00.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-08-19T14:13:00.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:13:00.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:13:00.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:13:00.127 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:13:00.127 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:13:00.127 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:13:00.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:13:00.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:13:00.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:13:00.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:13:00.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:13:00.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-08-19T14:13:00.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-08-19T14:13:00.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:13:00.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:13:00.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:13:00.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:13:00.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:13:00.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-08-19T14:13:00.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-08-19T14:13:00.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:13:00.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5a797ee4-e030-443f-b439-823062e8afc6 2024-08-19T14:13:00.129 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 5a797ee4-e030-443f-b439-823062e8afc6 2024-08-19T14:13:00.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 5a797ee4-e030-443f-b439-823062e8afc6' 2024-08-19T14:13:00.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:13:00.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDsUsNm8ZiICBAAs7f2bjN7X5bfNLg0SDrquQ== 2024-08-19T14:13:00.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDsUsNm8ZiICBAAs7f2bjN7X5bfNLg0SDrquQ=="}' 2024-08-19T14:13:00.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5a797ee4-e030-443f-b439-823062e8afc6 -i td/osd-recovery-scrub/3/new.json 2024-08-19T14:13:00.483 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:13:00.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-08-19T14:13:00.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQDsUsNm8ZiICBAAs7f2bjN7X5bfNLg0SDrquQ== --osd-uuid 5a797ee4-e030-443f-b439-823062e8afc6 2024-08-19T14:13:00.523 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:13:00.523+0000 7f061fa12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:13:00.568 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:13:00.568+0000 7f061fa12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:13:00.569 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:13:00.569+0000 7f061fa12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:13:00.570 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:13:00.569+0000 7f061fa12640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-08-19T14:13:00.570 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:13:00.569+0000 7f061fa12640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-08-19T14:13:00.570 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:13:00.569+0000 7f061fa12640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-08-19T14:13:00.643 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:13:00.643+0000 7f061fa12640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-08-19T14:13:06.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-08-19T14:13:06.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:13:06.704 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-08-19T14:13:06.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-08-19T14:13:06.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:13:07.052 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-08-19T14:13:07.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-08-19T14:13:07.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-08-19T14:13:07.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:13:07.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:13:07.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:13:07.079 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:13:07.079+0000 7fe7ca0e5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:13:07.081 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:13:07.081+0000 7fe7ca0e5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:13:07.082 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:13:07.082+0000 7fe7ca0e5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:13:07.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-08-19T14:13:07.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:13:07.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T14:13:07.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:13:07.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:13:07.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:13:07.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:13:07.318 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:13:07.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:13:07.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:13:07.599 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:13:07.599+0000 7fe7ca0e5640 -1 Falling back to public interface 2024-08-19T14:13:07.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:13:08.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:13:08.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:13:08.616 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:13:08.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:13:08.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:13:08.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:13:08.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:13:09.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:13:09.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:13:09.888 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:13:09.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:13:09.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:13:09.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:13:10.028 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:13:10.027+0000 7fe7ca0e5640 -1 osd.3 0 log_to_monitors true 2024-08-19T14:13:10.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:13:11.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:13:11.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:13:11.237 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:13:11.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:13:11.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:13:11.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:13:11.513 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3000872547,v1:127.0.0.1:6827/3000872547] [v2:127.0.0.1:6828/3000872547,v1:127.0.0.1:6829/3000872547] exists,up 5a797ee4-e030-443f-b439-823062e8afc6 2024-08-19T14:13:11.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:13:11.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:13:11.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:13:11.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2024-08-19T14:13:11.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-08-19T14:13:11.898 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' created 2024-08-19T14:13:11.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T14:13:12.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2024-08-19T14:13:12.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:13:12.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:13:12.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:13:12.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:13:12.914 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:13:12.914 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:13:12.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:13:12.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:13:12.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:13:13.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:13:13.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:13:13.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:13:13.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:13:13.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:13:13.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:13:13.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:13:13.298 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:13:13.298 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:13:13.298 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-08-19T14:13:13.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:13:13.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:13:13.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:13:13.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-08-19T14:13:13.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-08-19T14:13:13.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-08-19T14:13:13.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:13:13.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:13:13.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-08-19T14:13:13.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-08-19T14:13:13.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672966' 2024-08-19T14:13:13.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:13:13.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:13:13.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-08-19T14:13:13.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-08-19T14:13:13.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672966 2-64424509444' 2024-08-19T14:13:13.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:13:13.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:13:13.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345922 2024-08-19T14:13:13.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345922 2024-08-19T14:13:13.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672966 2-64424509444 3-85899345922' 2024-08-19T14:13:13.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:13:13.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-08-19T14:13:13.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:13:13.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:13:13.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-08-19T14:13:13.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:13:13.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-08-19T14:13:13.715 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836489 2024-08-19T14:13:13.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-08-19T14:13:13.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:13:13.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836489 2024-08-19T14:13:13.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:13:14.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:13:14.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:13:15.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836489 2024-08-19T14:13:15.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:13:16.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T14:13:16.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:13:16.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836489 2024-08-19T14:13:16.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:13:16.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-08-19T14:13:16.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:13:16.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:13:16.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-08-19T14:13:16.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:13:16.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-08-19T14:13:16.527 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672966 2024-08-19T14:13:16.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-08-19T14:13:16.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:13:16.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672966 2024-08-19T14:13:16.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:13:16.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-08-19T14:13:16.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:13:16.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:13:16.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-08-19T14:13:16.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:13:16.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-08-19T14:13:16.798 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509444 2024-08-19T14:13:16.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-08-19T14:13:16.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:13:17.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509444 2024-08-19T14:13:17.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:13:17.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345922 2024-08-19T14:13:17.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:13:17.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:13:17.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345922 2024-08-19T14:13:17.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:13:17.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345922 2024-08-19T14:13:17.071 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345922 2024-08-19T14:13:17.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2024-08-19T14:13:17.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:13:17.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345922 -lt 85899345922 2024-08-19T14:13:17.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:13:17.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:13:17.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:13:17.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:13:17.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:13:17.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:13:17.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:13:17.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:13:17.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:13:17.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:13:17.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:13:17.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:13:17.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:13:17.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:13:17.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:13:18.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:13:18.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:13:18.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:13:18.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2024-08-19T14:13:18.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2024-08-19T14:13:18.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2024-08-19T14:13:18.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2024-08-19T14:13:18.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2024-08-19T14:13:18.853 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T14:13:18.853 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:13:12.502430+0000 0'0 24:11 [1] 1 [1] 1 0'0 2024-08-19T14:13:11.855564+0000 0'0 2024-08-19T14:13:11.855564+0000 0 0 periodic scrub scheduled @ 2024-08-20T19:48:16.881545+0000 0 0 2024-08-19T14:13:18.854 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:13:18.854 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T14:13:18.854 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:13:18.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.32181 bs=1M count=50 2024-08-19T14:13:19.005 INFO:tasks.workunit.client.0.smithi102.stderr:50+0 records in 2024-08-19T14:13:19.005 INFO:tasks.workunit.client.0.smithi102.stderr:50+0 records out 2024-08-19T14:13:19.006 INFO:tasks.workunit.client.0.smithi102.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.139742 s, 375 MB/s 2024-08-19T14:13:19.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2024-08-19T14:13:19.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:19.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.32181 2024-08-19T14:13:20.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:20.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.32181 2024-08-19T14:13:22.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:22.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.32181 2024-08-19T14:13:23.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:23.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.32181 2024-08-19T14:13:25.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:25.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.32181 2024-08-19T14:13:26.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:26.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.32181 2024-08-19T14:13:28.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:28.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.32181 2024-08-19T14:13:29.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:29.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.32181 2024-08-19T14:13:31.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:31.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.32181 2024-08-19T14:13:33.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:33.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.32181 2024-08-19T14:13:34.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:34.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.32181 2024-08-19T14:13:36.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:36.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.32181 2024-08-19T14:13:37.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:37.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.32181 2024-08-19T14:13:39.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:39.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.32181 2024-08-19T14:13:40.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:40.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.32181 2024-08-19T14:13:42.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:42.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.32181 2024-08-19T14:13:43.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:43.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.32181 2024-08-19T14:13:44.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:44.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.32181 2024-08-19T14:13:46.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:46.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.32181 2024-08-19T14:13:47.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:47.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.32181 2024-08-19T14:13:49.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:49.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.32181 2024-08-19T14:13:50.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:50.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.32181 2024-08-19T14:13:52.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:52.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.32181 2024-08-19T14:13:54.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:54.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.32181 2024-08-19T14:13:55.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:55.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.32181 2024-08-19T14:13:57.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:57.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.32181 2024-08-19T14:13:58.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:13:58.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.32181 2024-08-19T14:14:00.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:00.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.32181 2024-08-19T14:14:01.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:01.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.32181 2024-08-19T14:14:02.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:02.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.32181 2024-08-19T14:14:04.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:04.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.32181 2024-08-19T14:14:05.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:05.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.32181 2024-08-19T14:14:07.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:07.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.32181 2024-08-19T14:14:08.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:08.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.32181 2024-08-19T14:14:10.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:10.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.32181 2024-08-19T14:14:11.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:11.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.32181 2024-08-19T14:14:13.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:13.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.32181 2024-08-19T14:14:14.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:14.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.32181 2024-08-19T14:14:16.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:16.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.32181 2024-08-19T14:14:17.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:17.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.32181 2024-08-19T14:14:19.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:19.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.32181 2024-08-19T14:14:20.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:20.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.32181 2024-08-19T14:14:22.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:22.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.32181 2024-08-19T14:14:23.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:23.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.32181 2024-08-19T14:14:25.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:25.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.32181 2024-08-19T14:14:26.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:26.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.32181 2024-08-19T14:14:27.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:27.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.32181 2024-08-19T14:14:29.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:29.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.32181 2024-08-19T14:14:30.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:30.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.32181 2024-08-19T14:14:32.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:32.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.32181 2024-08-19T14:14:33.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:33.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.32181 2024-08-19T14:14:35.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:35.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.32181 2024-08-19T14:14:36.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:36.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.32181 2024-08-19T14:14:38.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:38.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.32181 2024-08-19T14:14:39.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:39.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.32181 2024-08-19T14:14:41.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:41.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.32181 2024-08-19T14:14:42.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:42.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.32181 2024-08-19T14:14:44.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:44.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.32181 2024-08-19T14:14:45.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:45.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.32181 2024-08-19T14:14:47.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:47.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.32181 2024-08-19T14:14:48.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:48.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.32181 2024-08-19T14:14:50.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:50.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.32181 2024-08-19T14:14:51.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:51.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.32181 2024-08-19T14:14:52.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:52.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.32181 2024-08-19T14:14:54.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:54.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.32181 2024-08-19T14:14:55.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:55.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.32181 2024-08-19T14:14:57.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:57.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.32181 2024-08-19T14:14:58.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:14:58.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.32181 2024-08-19T14:15:00.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:00.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.32181 2024-08-19T14:15:01.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:01.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.32181 2024-08-19T14:15:03.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:03.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.32181 2024-08-19T14:15:04.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:04.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.32181 2024-08-19T14:15:06.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:06.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.32181 2024-08-19T14:15:07.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:07.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.32181 2024-08-19T14:15:09.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:09.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.32181 2024-08-19T14:15:10.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:10.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.32181 2024-08-19T14:15:12.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:12.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.32181 2024-08-19T14:15:13.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:13.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.32181 2024-08-19T14:15:15.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:15.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.32181 2024-08-19T14:15:16.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:16.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.32181 2024-08-19T14:15:18.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:18.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.32181 2024-08-19T14:15:19.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:19.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.32181 2024-08-19T14:15:21.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:21.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.32181 2024-08-19T14:15:22.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:22.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.32181 2024-08-19T14:15:24.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:24.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.32181 2024-08-19T14:15:25.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:25.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.32181 2024-08-19T14:15:26.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:26.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.32181 2024-08-19T14:15:28.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:28.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.32181 2024-08-19T14:15:29.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:29.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.32181 2024-08-19T14:15:31.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:31.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.32181 2024-08-19T14:15:32.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:32.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.32181 2024-08-19T14:15:34.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:34.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.32181 2024-08-19T14:15:35.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:35.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.32181 2024-08-19T14:15:37.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:37.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.32181 2024-08-19T14:15:38.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:38.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.32181 2024-08-19T14:15:40.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:40.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.32181 2024-08-19T14:15:41.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:41.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.32181 2024-08-19T14:15:43.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:43.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.32181 2024-08-19T14:15:44.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:44.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.32181 2024-08-19T14:15:46.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-08-19T14:15:46.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.32181 2024-08-19T14:15:47.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.32181 2024-08-19T14:15:47.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2024-08-19T14:15:48.182 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 4 2024-08-19T14:15:48.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2024-08-19T14:15:48.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2024-08-19T14:15:48.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-08-19T14:15:48.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-08-19T14:15:48.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-08-19T14:15:48.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-08-19T14:15:48.447 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:15:48.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-08-19T14:15:50.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2024-08-19T14:15:50.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2024-08-19T14:15:50.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2024-08-19T14:15:50.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-08-19T14:15:50.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-08-19T14:15:50.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-08-19T14:15:50.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-08-19T14:15:50.724 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:15:50.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-08-19T14:15:52.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2024-08-19T14:15:52.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2024-08-19T14:15:52.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2024-08-19T14:15:52.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-08-19T14:15:52.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-08-19T14:15:52.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-08-19T14:15:52.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-08-19T14:15:52.998 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:15:53.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-08-19T14:15:55.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 2 -eq 10 2024-08-19T14:15:55.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 2 + 1 2024-08-19T14:15:55.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=3 2024-08-19T14:15:55.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-08-19T14:15:55.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-08-19T14:15:55.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-08-19T14:15:55.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-08-19T14:15:55.272 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:15:55.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-08-19T14:15:57.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 3 -eq 10 2024-08-19T14:15:57.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 3 + 1 2024-08-19T14:15:57.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=4 2024-08-19T14:15:57.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-08-19T14:15:57.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-08-19T14:15:57.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-08-19T14:15:57.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-08-19T14:15:57.550 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:15:57.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2024-08-19T14:15:57.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2024-08-19T14:15:57.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2024-08-19T14:15:57.931 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T14:15:57.931 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 297 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2024-08-19T14:15:50.087867+0000 24'1300 28:1328 [1,0,2,3] 1 [1,0] 1 0'0 2024-08-19T14:13:11.855564+0000 0'0 2024-08-19T14:13:11.855564+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:15:57.931 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:15:57.931 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T14:15:57.931 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:15:57.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2024-08-19T14:16:07.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2024-08-19T14:16:07.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:16:07.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:16:07.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:16:07.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:16:07.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:16:09.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:16:09.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2024-08-19T14:16:09.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress. Only high priority scrubs allowed.' 2024-08-19T14:16:09.255 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2024-08-19T14:16:09.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2024-08-19T14:16:09.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:16:09.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-08-19T14:16:09.260 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:15:54.358+0000 7f0c36fe2640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-08-19T14:16:09.260 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:15:55.317+0000 7f0c36fe2640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-08-19T14:16:09.261 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:16:05.343+0000 7f0c36fe2640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-08-19T14:16:09.261 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:16:07.255+0000 7f0c36fe2640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-08-19T14:16:09.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:16:09.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-08-19T14:16:09.273 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:15:50.199+0000 7f7c15b77640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-08-19T14:16:09.274 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:15:53.185+0000 7f7c15b77640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-08-19T14:16:09.274 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:15:56.141+0000 7f7c15b77640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-08-19T14:16:09.274 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:15:57.129+0000 7f7c15b77640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-08-19T14:16:09.274 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:15:59.158+0000 7f7c15b77640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-08-19T14:16:09.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:16:09.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-08-19T14:16:09.276 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:16:00.739+0000 7f481b12b640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-08-19T14:16:09.276 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:16:03.722+0000 7f481b12b640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-08-19T14:16:09.277 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:16:07.766+0000 7f481b12b640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-08-19T14:16:09.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:16:09.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-08-19T14:16:09.279 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:16:01.930+0000 7fe7c3088640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-08-19T14:16:09.279 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:16:02.911+0000 7fe7c3088640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-08-19T14:16:09.279 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:16:03.866+0000 7fe7c3088640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-08-19T14:16:09.279 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:16:04.884+0000 7fe7c3088640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-08-19T14:16:09.279 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:16:05.846+0000 7fe7c3088640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-08-19T14:16:09.280 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:16:07.788+0000 7fe7c3088640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-08-19T14:16:09.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2024-08-19T14:16:09.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2024-08-19T14:16:09.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2024-08-19T14:16:09.281 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2024-08-19T14:16:09.281 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2024-08-19T14:16:09.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:16:09.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-08-19T14:16:09.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-08-19T14:16:09.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2024-08-19T14:16:09.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2024-08-19T14:16:09.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:16:09.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-08-19T14:16:09.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-08-19T14:16:09.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2024-08-19T14:16:09.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2024-08-19T14:16:09.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:16:09.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-08-19T14:16:09.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-08-19T14:16:09.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2024-08-19T14:16:09.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2024-08-19T14:16:09.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:16:09.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-08-19T14:16:09.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-08-19T14:16:09.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2024-08-19T14:16:09.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2024-08-19T14:16:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2024-08-19T14:16:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2024-08-19T14:16:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2024-08-19T14:16:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-08-19T14:16:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:16:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-08-19T14:16:09.308 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:16:09.308 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:16:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:16:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:16:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:16:09.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:16:09.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:16:09.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:16:09.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:16:09.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:16:09.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:16:09.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:16:09.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:16:09.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:16:09.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:16:09.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:16:09.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:16:09.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:16:09.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-08-19T14:16:09.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:16:09.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:16:09.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:16:09.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32181 2024-08-19T14:16:09.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:16:09.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:16:09.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2024-08-19T14:16:09.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2024-08-19T14:16:09.343 INFO:tasks.workunit.client.0.smithi102.stdout:TEST PASSED 2024-08-19T14:16:09.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2024-08-19T14:16:09.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-08-19T14:16:09.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2024-08-19T14:16:09.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2024-08-19T14:16:09.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: local poolname=test 2024-08-19T14:16:09.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:237: TEST_recovery_scrub_2: TESTDATA=testdata.32181 2024-08-19T14:16:09.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: OSDS=8 2024-08-19T14:16:09.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: PGS=32 2024-08-19T14:16:09.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:240: TEST_recovery_scrub_2: OBJECTS=40 2024-08-19T14:16:09.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2024-08-19T14:16:09.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-08-19T14:16:09.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-08-19T14:16:09.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-08-19T14:16:09.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:16:09.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-08-19T14:16:09.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:16:09.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:16:09.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:16:09.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:16:09.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:16:09.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:16:09.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:16:09.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:16:09.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:16:09.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:16:09.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:16:09.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:16:09.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:16:09.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:16:09.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:16:09.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:16:09.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:16:09.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:16:09.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-08-19T14:16:09.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:16:09.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:16:09.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:16:09.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32181 2024-08-19T14:16:09.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:16:09.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:16:09.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-08-19T14:16:09.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T14:16:09.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:16:09.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:16:09.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.32181 2024-08-19T14:16:09.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T14:16:09.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T14:16:09.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T14:16:09.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-08-19T14:16:09.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:243: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T14:16:09.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-08-19T14:16:09.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T14:16:09.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T14:16:09.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T14:16:09.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-08-19T14:16:09.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T14:16:09.742 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T14:16:09.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:16:09.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:16:09.746 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:16:09.746 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:16:09.746 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:16:09.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:16:09.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T14:16:09.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T14:16:09.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T14:16:09.928 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:16:09.928 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:16:09.928 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T14:16:09.928 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T14:16:09.928 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:16:09.928 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:16:09.928 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:16:09.929 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:16:09.929 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:16:09.929 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:16:09.929 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32181/ceph-mon.a.asok 2024-08-19T14:16:09.929 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:16:09.929 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32181/ceph-mon.a.asok config get fsid 2024-08-19T14:16:10.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T14:16:10.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:16:10.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:16:10.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T14:16:10.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T14:16:10.009 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:16:10.009 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:16:10.010 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:16:10.010 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:16:10.010 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:16:10.010 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:16:10.010 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32181/ceph-mon.a.asok 2024-08-19T14:16:10.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:16:10.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32181/ceph-mon.a.asok config get mon_host 2024-08-19T14:16:10.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2024-08-19T14:16:10.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-08-19T14:16:10.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T14:16:10.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T14:16:10.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T14:16:10.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-08-19T14:16:10.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T14:16:10.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T14:16:10.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:16:10.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:16:10.895 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:16:10.895 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:16:10.895 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:16:10.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:16:10.896 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:16:10.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:16:10.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:245: TEST_recovery_scrub_2: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-08-19T14:16:10.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:246: TEST_recovery_scrub_2: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-08-19T14:16:10.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:247: TEST_recovery_scrub_2: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-08-19T14:16:10.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-08-19T14:16:10.923 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: expr 8 - 1 2024-08-19T14:16:10.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: seq 0 7 2024-08-19T14:16:10.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:16:10.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-08-19T14:16:10.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-08-19T14:16:10.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:16:10.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T14:16:10.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:16:10.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-08-19T14:16:10.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-08-19T14:16:10.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:16:10.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:16:10.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:16:10.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-08-19T14:16:10.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-08-19T14:16:10.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:16:10.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:16:10.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-08-19T14:16:10.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:16:10.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:16:10.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:16:10.931 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:16:10.931 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:16:10.931 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:16:10.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:16:10.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:16:10.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:16:10.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:16:10.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:16:10.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-08-19T14:16:10.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-08-19T14:16:10.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:16:10.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:16:10.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:16:10.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:16:10.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:16:10.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-08-19T14:16:10.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-08-19T14:16:10.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:16:10.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=72d7e2da-f712-49eb-9ed1-23badba0d6df 2024-08-19T14:16:10.933 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 72d7e2da-f712-49eb-9ed1-23badba0d6df 2024-08-19T14:16:10.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 72d7e2da-f712-49eb-9ed1-23badba0d6df' 2024-08-19T14:16:10.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:16:10.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCqU8Nm0FtkOBAA6QpfeSLrGShB4uyaIQ46+g== 2024-08-19T14:16:10.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCqU8Nm0FtkOBAA6QpfeSLrGShB4uyaIQ46+g=="}' 2024-08-19T14:16:10.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 72d7e2da-f712-49eb-9ed1-23badba0d6df -i td/osd-recovery-scrub/0/new.json 2024-08-19T14:16:11.143 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:16:11.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-08-19T14:16:11.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCqU8Nm0FtkOBAA6QpfeSLrGShB4uyaIQ46+g== --osd-uuid 72d7e2da-f712-49eb-9ed1-23badba0d6df 2024-08-19T14:16:11.185 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:11.185+0000 7fc0fd0b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:11.193 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:11.193+0000 7fc0fd0b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:11.195 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:11.194+0000 7fc0fd0b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:11.195 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:11.195+0000 7fc0fd0b5640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-08-19T14:16:11.195 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:11.195+0000 7fc0fd0b5640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-08-19T14:16:11.195 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:11.195+0000 7fc0fd0b5640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-08-19T14:16:11.277 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:11.277+0000 7fc0fd0b5640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-08-19T14:16:16.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-08-19T14:16:16.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:16:16.781 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T14:16:16.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T14:16:16.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:16:17.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T14:16:17.128 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T14:16:17.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-08-19T14:16:17.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:16:17.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:16:17.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:16:17.159 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:17.158+0000 7f9ac2e24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:17.160 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:17.160+0000 7f9ac2e24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:17.161 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:17.161+0000 7f9ac2e24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:17.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T14:16:17.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:16:17.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:16:17.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:16:17.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:16:17.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:16:17.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:16:17.395 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:16:17.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:16:17.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:16:17.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:16:17.688 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:17.688+0000 7f9ac2e24640 -1 Falling back to public interface 2024-08-19T14:16:18.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:16:18.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:16:18.659 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:16:18.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:16:18.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:16:18.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:16:18.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:16:19.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:16:19.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:16:19.925 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:16:19.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:16:19.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:16:19.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:16:20.052 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:20.051+0000 7f9ac2e24640 -1 osd.0 0 log_to_monitors true 2024-08-19T14:16:20.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:16:21.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:16:21.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:16:21.227 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:16:21.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:16:21.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:16:21.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:16:21.565 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2673339464,v1:127.0.0.1:6803/2673339464] [v2:127.0.0.1:6804/2673339464,v1:127.0.0.1:6805/2673339464] exists,up 72d7e2da-f712-49eb-9ed1-23badba0d6df 2024-08-19T14:16:21.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:16:21.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:16:21.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:16:21.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:16:21.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-08-19T14:16:21.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-08-19T14:16:21.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:16:21.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T14:16:21.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:16:21.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-08-19T14:16:21.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-08-19T14:16:21.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:16:21.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:16:21.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:16:21.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-08-19T14:16:21.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-08-19T14:16:21.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:16:21.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:16:21.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-08-19T14:16:21.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:16:21.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:16:21.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:16:21.571 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:16:21.571 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:16:21.571 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:16:21.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:16:21.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:16:21.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:16:21.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:16:21.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:16:21.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-08-19T14:16:21.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-08-19T14:16:21.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:16:21.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:16:21.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:16:21.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:16:21.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:16:21.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-08-19T14:16:21.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-08-19T14:16:21.573 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:16:21.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2bc09357-42cc-46c1-b6e6-cd14aa2bf9e2 2024-08-19T14:16:21.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2bc09357-42cc-46c1-b6e6-cd14aa2bf9e2' 2024-08-19T14:16:21.573 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 2bc09357-42cc-46c1-b6e6-cd14aa2bf9e2 2024-08-19T14:16:21.573 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:16:21.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC1U8NmbW70IhAAz+ktJ6TR0ePP2LHqGLccTw== 2024-08-19T14:16:21.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC1U8NmbW70IhAAz+ktJ6TR0ePP2LHqGLccTw=="}' 2024-08-19T14:16:21.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2bc09357-42cc-46c1-b6e6-cd14aa2bf9e2 -i td/osd-recovery-scrub/1/new.json 2024-08-19T14:16:21.954 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:16:21.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-08-19T14:16:21.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQC1U8NmbW70IhAAz+ktJ6TR0ePP2LHqGLccTw== --osd-uuid 2bc09357-42cc-46c1-b6e6-cd14aa2bf9e2 2024-08-19T14:16:21.999 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:21.998+0000 7f95cac0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:22.039 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:22.039+0000 7f95cac0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:22.040 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:22.040+0000 7f95cac0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:22.040 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:22.040+0000 7f95cac0f640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-08-19T14:16:22.041 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:22.040+0000 7f95cac0f640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-08-19T14:16:22.041 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:22.040+0000 7f95cac0f640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-08-19T14:16:22.124 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:22.124+0000 7f95cac0f640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-08-19T14:16:27.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-08-19T14:16:27.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:16:27.954 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T14:16:27.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T14:16:27.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:16:28.299 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T14:16:28.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T14:16:28.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-08-19T14:16:28.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:16:28.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:16:28.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:16:28.330 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:28.330+0000 7f7db7a5f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:28.333 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:28.332+0000 7f7db7a5f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:28.333 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:28.333+0000 7f7db7a5f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:28.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T14:16:28.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:16:28.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:16:28.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:16:28.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:16:28.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:16:28.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:16:28.568 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:16:28.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:16:28.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:16:28.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:16:28.855 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:28.855+0000 7f7db7a5f640 -1 Falling back to public interface 2024-08-19T14:16:29.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:16:29.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:16:29.834 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:16:29.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:16:29.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:16:29.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:16:30.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:16:31.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:16:31.106 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:16:31.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:16:31.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:16:31.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:16:31.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:16:31.364 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:31.362+0000 7f7db7a5f640 -1 osd.1 0 log_to_monitors true 2024-08-19T14:16:31.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:16:32.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:16:32.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:16:32.409 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:16:32.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:16:32.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:16:32.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:16:32.712 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2965312871,v1:127.0.0.1:6811/2965312871] [v2:127.0.0.1:6812/2965312871,v1:127.0.0.1:6813/2965312871] exists,up 2bc09357-42cc-46c1-b6e6-cd14aa2bf9e2 2024-08-19T14:16:32.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:16:32.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:16:32.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:16:32.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:16:32.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-08-19T14:16:32.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-08-19T14:16:32.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:16:32.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T14:16:32.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:16:32.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-08-19T14:16:32.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-08-19T14:16:32.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:16:32.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:16:32.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:16:32.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-08-19T14:16:32.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-08-19T14:16:32.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:16:32.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:16:32.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-08-19T14:16:32.723 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:16:32.723 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:16:32.723 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:16:32.724 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:16:32.724 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:16:32.724 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:16:32.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:16:32.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:16:32.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:16:32.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:16:32.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:16:32.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-08-19T14:16:32.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-08-19T14:16:32.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:16:32.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:16:32.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:16:32.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:16:32.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:16:32.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-08-19T14:16:32.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-08-19T14:16:32.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:16:32.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=814a0f8d-723f-448c-bba4-1e76fa3db16e 2024-08-19T14:16:32.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 814a0f8d-723f-448c-bba4-1e76fa3db16e' 2024-08-19T14:16:32.728 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 814a0f8d-723f-448c-bba4-1e76fa3db16e 2024-08-19T14:16:32.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:16:32.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDAU8NmMAS8KxAAsxiK0kMDe0P39YbvxNkJDw== 2024-08-19T14:16:32.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDAU8NmMAS8KxAAsxiK0kMDe0P39YbvxNkJDw=="}' 2024-08-19T14:16:32.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 814a0f8d-723f-448c-bba4-1e76fa3db16e -i td/osd-recovery-scrub/2/new.json 2024-08-19T14:16:33.143 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:16:33.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-08-19T14:16:33.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDAU8NmMAS8KxAAsxiK0kMDe0P39YbvxNkJDw== --osd-uuid 814a0f8d-723f-448c-bba4-1e76fa3db16e 2024-08-19T14:16:33.187 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:33.186+0000 7f881cc24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:33.227 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:33.226+0000 7f881cc24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:33.228 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:33.227+0000 7f881cc24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:33.229 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:33.227+0000 7f881cc24640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-08-19T14:16:33.229 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:33.227+0000 7f881cc24640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-08-19T14:16:33.229 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:33.227+0000 7f881cc24640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-08-19T14:16:33.303 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:33.302+0000 7f881cc24640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-08-19T14:16:39.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-08-19T14:16:39.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:16:39.238 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T14:16:39.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T14:16:39.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:16:39.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T14:16:39.584 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T14:16:39.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-08-19T14:16:39.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:16:39.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:16:39.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:16:39.615 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:39.615+0000 7f16f341a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:39.617 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:39.617+0000 7f16f341a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:39.619 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:39.619+0000 7f16f341a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:39.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T14:16:39.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:16:39.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T14:16:39.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:16:39.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:16:39.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:16:39.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:16:39.852 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:16:39.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:16:39.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:16:40.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:16:40.146 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:40.146+0000 7f16f341a640 -1 Falling back to public interface 2024-08-19T14:16:41.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:16:41.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:16:41.118 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:16:41.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:16:41.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:16:41.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:16:41.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:16:42.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:16:42.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:16:42.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:16:42.394 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:16:42.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:16:42.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:16:42.611 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:42.611+0000 7f16f341a640 -1 osd.2 0 log_to_monitors true 2024-08-19T14:16:42.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:16:43.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:16:43.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:16:43.669 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:16:43.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:16:43.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:16:43.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:16:43.947 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3750666973,v1:127.0.0.1:6819/3750666973] [v2:127.0.0.1:6820/3750666973,v1:127.0.0.1:6821/3750666973] exists,up 814a0f8d-723f-448c-bba4-1e76fa3db16e 2024-08-19T14:16:43.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:16:43.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:16:43.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:16:43.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:16:43.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-08-19T14:16:43.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-08-19T14:16:43.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:16:43.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-08-19T14:16:43.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:16:43.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-08-19T14:16:43.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-08-19T14:16:43.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:16:43.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:16:43.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:16:43.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-08-19T14:16:43.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-08-19T14:16:43.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:16:43.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:16:43.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-08-19T14:16:43.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:16:43.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:16:43.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:16:43.952 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:16:43.953 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:16:43.953 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:16:43.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:16:43.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:16:43.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:16:43.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:16:43.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:16:43.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-08-19T14:16:43.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-08-19T14:16:43.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:16:43.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:16:43.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:16:43.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:16:43.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:16:43.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-08-19T14:16:43.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-08-19T14:16:43.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:16:43.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2221b7bc-95b0-4248-b83a-08ca5e57d12b 2024-08-19T14:16:43.954 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 2221b7bc-95b0-4248-b83a-08ca5e57d12b 2024-08-19T14:16:43.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 2221b7bc-95b0-4248-b83a-08ca5e57d12b' 2024-08-19T14:16:43.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:16:43.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLU8NmbyPLORAA+6w9rCO3S+4HZ0Zjcre8GQ== 2024-08-19T14:16:43.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLU8NmbyPLORAA+6w9rCO3S+4HZ0Zjcre8GQ=="}' 2024-08-19T14:16:43.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2221b7bc-95b0-4248-b83a-08ca5e57d12b -i td/osd-recovery-scrub/3/new.json 2024-08-19T14:16:44.307 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:16:44.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-08-19T14:16:44.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDLU8NmbyPLORAA+6w9rCO3S+4HZ0Zjcre8GQ== --osd-uuid 2221b7bc-95b0-4248-b83a-08ca5e57d12b 2024-08-19T14:16:44.351 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:44.350+0000 7f215801d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:44.391 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:44.391+0000 7f215801d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:44.392 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:44.392+0000 7f215801d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:44.393 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:44.392+0000 7f215801d640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-08-19T14:16:44.393 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:44.392+0000 7f215801d640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-08-19T14:16:44.393 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:44.392+0000 7f215801d640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-08-19T14:16:44.466 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:44.466+0000 7f215801d640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-08-19T14:16:50.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-08-19T14:16:50.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:16:50.286 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-08-19T14:16:50.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-08-19T14:16:50.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:16:50.630 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-08-19T14:16:50.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-08-19T14:16:50.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-08-19T14:16:50.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:16:50.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:16:50.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:16:50.661 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:50.660+0000 7fdc49e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:50.662 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:50.662+0000 7fdc49e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:50.663 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:50.663+0000 7fdc49e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:50.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-08-19T14:16:50.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:16:50.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T14:16:50.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:16:50.896 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:16:50.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:16:50.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:16:50.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:16:50.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:16:50.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:16:51.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:16:51.185 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:51.185+0000 7fdc49e0f640 -1 Falling back to public interface 2024-08-19T14:16:52.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:16:52.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:16:52.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:16:52.161 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:16:52.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:16:52.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:16:52.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:16:53.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:16:53.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:16:53.434 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:16:53.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:16:53.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:16:53.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:16:53.558 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:53.558+0000 7fdc49e0f640 -1 osd.3 0 log_to_monitors true 2024-08-19T14:16:53.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:16:54.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:16:54.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:16:54.768 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:16:54.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:16:54.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:16:54.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:16:55.062 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2666136377,v1:127.0.0.1:6827/2666136377] [v2:127.0.0.1:6828/2666136377,v1:127.0.0.1:6829/2666136377] exists,up 2221b7bc-95b0-4248-b83a-08ca5e57d12b 2024-08-19T14:16:55.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:16:55.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:16:55.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:16:55.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:16:55.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-08-19T14:16:55.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-08-19T14:16:55.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:16:55.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-08-19T14:16:55.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:16:55.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/4 2024-08-19T14:16:55.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-08-19T14:16:55.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:16:55.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:16:55.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:16:55.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2024-08-19T14:16:55.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2024-08-19T14:16:55.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:16:55.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:16:55.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-08-19T14:16:55.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:16:55.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:16:55.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:16:55.068 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:16:55.068 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:16:55.068 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:16:55.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:16:55.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:16:55.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:16:55.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:16:55.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:16:55.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-08-19T14:16:55.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-08-19T14:16:55.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:16:55.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:16:55.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:16:55.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:16:55.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:16:55.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-08-19T14:16:55.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/4 2024-08-19T14:16:55.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:16:55.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b97617e6-27fc-4859-9c39-09e4a319a44e 2024-08-19T14:16:55.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 b97617e6-27fc-4859-9c39-09e4a319a44e' 2024-08-19T14:16:55.070 INFO:tasks.workunit.client.0.smithi102.stdout:add osd4 b97617e6-27fc-4859-9c39-09e4a319a44e 2024-08-19T14:16:55.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:16:55.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDXU8Nmw7P/BBAABwq0ZVUde6HCEya1cqfSnA== 2024-08-19T14:16:55.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDXU8Nmw7P/BBAABwq0ZVUde6HCEya1cqfSnA=="}' 2024-08-19T14:16:55.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b97617e6-27fc-4859-9c39-09e4a319a44e -i td/osd-recovery-scrub/4/new.json 2024-08-19T14:16:55.430 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:16:55.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/4/new.json 2024-08-19T14:16:55.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDXU8Nmw7P/BBAABwq0ZVUde6HCEya1cqfSnA== --osd-uuid b97617e6-27fc-4859-9c39-09e4a319a44e 2024-08-19T14:16:55.475 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:55.475+0000 7fac3bdcb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:55.523 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:55.523+0000 7fac3bdcb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:55.524 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:55.524+0000 7fac3bdcb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:16:55.525 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:55.524+0000 7fac3bdcb640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-08-19T14:16:55.525 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:55.524+0000 7fac3bdcb640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-08-19T14:16:55.525 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:55.524+0000 7fac3bdcb640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-08-19T14:16:55.598 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:16:55.598+0000 7fac3bdcb640 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2024-08-19T14:17:01.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2024-08-19T14:17:01.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:17:01.286 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd4 key to auth repository 2024-08-19T14:17:01.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-08-19T14:17:01.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:17:01.634 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.4 2024-08-19T14:17:01.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-08-19T14:17:01.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-08-19T14:17:01.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:17:01.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:17:01.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:17:01.665 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:01.665+0000 7f8ee9dd8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:01.667 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:01.667+0000 7f8ee9dd8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:01.668 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:01.668+0000 7f8ee9dd8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:01.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-08-19T14:17:01.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:17:01.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-08-19T14:17:01.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:17:01.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:17:01.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:17:01.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:17:01.906 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:17:01.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:17:01.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:17:02.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:17:02.197 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:02.197+0000 7f8ee9dd8640 -1 Falling back to public interface 2024-08-19T14:17:03.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:17:03.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:17:03.169 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:17:03.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:17:03.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:17:03.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:17:03.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:17:04.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:17:04.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:17:04.447 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:17:04.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:17:04.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:17:04.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:17:04.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:04.532+0000 7f8ee9dd8640 -1 osd.4 0 log_to_monitors true 2024-08-19T14:17:04.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:17:05.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:17:05.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:17:05.726 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:17:05.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:17:05.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:17:05.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:17:05.999 INFO:tasks.workunit.client.0.smithi102.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1984285288,v1:127.0.0.1:6835/1984285288] [v2:127.0.0.1:6836/1984285288,v1:127.0.0.1:6837/1984285288] exists,up b97617e6-27fc-4859-9c39-09e4a319a44e 2024-08-19T14:17:06.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:17:06.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:17:06.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:17:06.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:17:06.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-08-19T14:17:06.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-08-19T14:17:06.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:17:06.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-08-19T14:17:06.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:17:06.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/5 2024-08-19T14:17:06.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-08-19T14:17:06.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:17:06.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:17:06.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:17:06.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2024-08-19T14:17:06.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2024-08-19T14:17:06.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:17:06.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:17:06.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-08-19T14:17:06.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:17:06.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:17:06.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:17:06.008 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:17:06.008 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:17:06.008 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:17:06.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:17:06.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:17:06.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:17:06.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:17:06.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:17:06.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-08-19T14:17:06.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-08-19T14:17:06.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:17:06.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:17:06.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:17:06.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:17:06.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:17:06.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-08-19T14:17:06.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/5 2024-08-19T14:17:06.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:17:06.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=757d81de-5920-4171-bc08-a696095f8485 2024-08-19T14:17:06.010 INFO:tasks.workunit.client.0.smithi102.stdout:add osd5 757d81de-5920-4171-bc08-a696095f8485 2024-08-19T14:17:06.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 757d81de-5920-4171-bc08-a696095f8485' 2024-08-19T14:17:06.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:17:06.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDiU8NmoX8zARAA8dvJNszlQ2bR/jL462WD5Q== 2024-08-19T14:17:06.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDiU8NmoX8zARAA8dvJNszlQ2bR/jL462WD5Q=="}' 2024-08-19T14:17:06.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 757d81de-5920-4171-bc08-a696095f8485 -i td/osd-recovery-scrub/5/new.json 2024-08-19T14:17:06.360 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:17:06.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/5/new.json 2024-08-19T14:17:06.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDiU8NmoX8zARAA8dvJNszlQ2bR/jL462WD5Q== --osd-uuid 757d81de-5920-4171-bc08-a696095f8485 2024-08-19T14:17:06.405 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:06.404+0000 7fd61b357640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:06.445 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:06.444+0000 7fd61b357640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:06.446 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:06.445+0000 7fd61b357640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:06.446 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:06.446+0000 7fd61b357640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-08-19T14:17:06.446 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:06.446+0000 7fd61b357640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-08-19T14:17:06.446 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:06.446+0000 7fd61b357640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-08-19T14:17:06.517 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:06.516+0000 7fd61b357640 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2024-08-19T14:17:12.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2024-08-19T14:17:12.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:17:12.705 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd5 key to auth repository 2024-08-19T14:17:12.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-08-19T14:17:12.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:17:13.057 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.5 2024-08-19T14:17:13.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-08-19T14:17:13.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-08-19T14:17:13.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:17:13.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:17:13.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:17:13.088 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:13.088+0000 7f06b720e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:13.115 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:13.115+0000 7f06b720e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:13.116 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:13.116+0000 7f06b720e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:13.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-08-19T14:17:13.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:17:13.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-08-19T14:17:13.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:17:13.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:17:13.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:17:13.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:17:13.327 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:17:13.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:17:13.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:17:13.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:17:13.633 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:13.633+0000 7f06b720e640 -1 Falling back to public interface 2024-08-19T14:17:14.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:17:14.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:17:14.607 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:17:14.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:17:14.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:17:14.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:17:14.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:17:15.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:17:15.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:17:15.884 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:17:15.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:17:15.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:17:15.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:17:15.974 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:15.973+0000 7f06b720e640 -1 osd.5 0 log_to_monitors true 2024-08-19T14:17:16.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:17:17.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:17:17.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:17:17.183 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:17:17.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:17:17.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:17:17.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:17:17.561 INFO:tasks.workunit.client.0.smithi102.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3312169817,v1:127.0.0.1:6843/3312169817] [v2:127.0.0.1:6844/3312169817,v1:127.0.0.1:6845/3312169817] exists,up 757d81de-5920-4171-bc08-a696095f8485 2024-08-19T14:17:17.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:17:17.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:17:17.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:17:17.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:17:17.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-08-19T14:17:17.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-08-19T14:17:17.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:17:17.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-08-19T14:17:17.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:17:17.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/6 2024-08-19T14:17:17.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-08-19T14:17:17.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:17:17.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:17:17.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:17:17.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2024-08-19T14:17:17.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2024-08-19T14:17:17.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:17:17.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:17:17.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-08-19T14:17:17.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:17:17.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:17:17.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:17:17.566 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:17:17.566 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:17:17.566 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:17:17.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:17:17.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:17:17.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:17:17.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:17:17.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:17:17.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-08-19T14:17:17.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-08-19T14:17:17.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:17:17.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:17:17.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:17:17.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:17:17.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:17:17.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-08-19T14:17:17.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/6 2024-08-19T14:17:17.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:17:17.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7b4aff85-35c3-44a2-882d-f77ecbed8f88 2024-08-19T14:17:17.568 INFO:tasks.workunit.client.0.smithi102.stdout:add osd6 7b4aff85-35c3-44a2-882d-f77ecbed8f88 2024-08-19T14:17:17.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 7b4aff85-35c3-44a2-882d-f77ecbed8f88' 2024-08-19T14:17:17.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:17:17.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDtU8Nm/Qa1IhAA/Ks4BJrwOOFJFHtSEOLrpQ== 2024-08-19T14:17:17.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDtU8Nm/Qa1IhAA/Ks4BJrwOOFJFHtSEOLrpQ=="}' 2024-08-19T14:17:17.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7b4aff85-35c3-44a2-882d-f77ecbed8f88 -i td/osd-recovery-scrub/6/new.json 2024-08-19T14:17:18.032 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:17:18.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/6/new.json 2024-08-19T14:17:18.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDtU8Nm/Qa1IhAA/Ks4BJrwOOFJFHtSEOLrpQ== --osd-uuid 7b4aff85-35c3-44a2-882d-f77ecbed8f88 2024-08-19T14:17:18.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:18.077+0000 7f39ddc14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:18.125 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:18.125+0000 7f39ddc14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:18.126 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:18.126+0000 7f39ddc14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:18.126 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:18.126+0000 7f39ddc14640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-08-19T14:17:18.127 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:18.126+0000 7f39ddc14640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-08-19T14:17:18.127 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:18.126+0000 7f39ddc14640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-08-19T14:17:18.242 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:18.242+0000 7f39ddc14640 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2024-08-19T14:17:24.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2024-08-19T14:17:24.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:17:24.283 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd6 key to auth repository 2024-08-19T14:17:24.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-08-19T14:17:24.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:17:24.626 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.6 2024-08-19T14:17:24.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-08-19T14:17:24.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-08-19T14:17:24.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:17:24.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:17:24.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:17:24.658 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:24.657+0000 7f0c15d57640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:24.659 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:24.659+0000 7f0c15d57640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:24.660 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:24.660+0000 7f0c15d57640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:24.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-08-19T14:17:24.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:17:24.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-08-19T14:17:24.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:17:24.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:17:24.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:17:24.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:17:24.895 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:17:24.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:17:24.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:17:25.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:17:25.192 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:25.191+0000 7f0c15d57640 -1 Falling back to public interface 2024-08-19T14:17:26.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:17:26.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:17:26.163 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:17:26.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:17:26.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:17:26.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:17:26.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:17:27.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:17:27.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:17:27.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:17:27.437 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:17:27.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:17:27.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:17:27.551 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:27.551+0000 7f0c15d57640 -1 osd.6 0 log_to_monitors true 2024-08-19T14:17:27.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:17:28.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:17:28.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:17:28.753 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:17:28.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:17:28.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:17:28.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:17:29.066 INFO:tasks.workunit.client.0.smithi102.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/4053655197,v1:127.0.0.1:6851/4053655197] [v2:127.0.0.1:6852/4053655197,v1:127.0.0.1:6853/4053655197] exists,up 7b4aff85-35c3-44a2-882d-f77ecbed8f88 2024-08-19T14:17:29.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:17:29.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:17:29.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:17:29.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:17:29.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-08-19T14:17:29.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-08-19T14:17:29.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:17:29.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-08-19T14:17:29.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:17:29.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/7 2024-08-19T14:17:29.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-08-19T14:17:29.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:17:29.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:17:29.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:17:29.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2024-08-19T14:17:29.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2024-08-19T14:17:29.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:17:29.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:17:29.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-08-19T14:17:29.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:17:29.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:17:29.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:17:29.073 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:17:29.073 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:17:29.073 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:17:29.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:17:29.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' 2024-08-19T14:17:29.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:17:29.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:17:29.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:17:29.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-08-19T14:17:29.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-08-19T14:17:29.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:17:29.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:17:29.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:17:29.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:17:29.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:17:29.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-08-19T14:17:29.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/7 2024-08-19T14:17:29.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:17:29.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=81618b62-18ef-4bc3-a68e-020802a73019 2024-08-19T14:17:29.074 INFO:tasks.workunit.client.0.smithi102.stdout:add osd7 81618b62-18ef-4bc3-a68e-020802a73019 2024-08-19T14:17:29.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 81618b62-18ef-4bc3-a68e-020802a73019' 2024-08-19T14:17:29.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:17:29.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD5U8NmqdhJBRAA0cRoAkWDpvK90wjwOPCcKA== 2024-08-19T14:17:29.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD5U8NmqdhJBRAA0cRoAkWDpvK90wjwOPCcKA=="}' 2024-08-19T14:17:29.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 81618b62-18ef-4bc3-a68e-020802a73019 -i td/osd-recovery-scrub/7/new.json 2024-08-19T14:17:29.483 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:17:29.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/7/new.json 2024-08-19T14:17:29.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQD5U8NmqdhJBRAA0cRoAkWDpvK90wjwOPCcKA== --osd-uuid 81618b62-18ef-4bc3-a68e-020802a73019 2024-08-19T14:17:29.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:29.528+0000 7fdb766b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:29.535 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:29.534+0000 7fdb766b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:29.536 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:29.536+0000 7fdb766b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:29.536 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:29.536+0000 7fdb766b1640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-08-19T14:17:29.537 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:29.536+0000 7fdb766b1640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-08-19T14:17:29.537 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:29.536+0000 7fdb766b1640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-08-19T14:17:29.635 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:29.634+0000 7fdb766b1640 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2024-08-19T14:17:35.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2024-08-19T14:17:35.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:17:35.684 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd7 key to auth repository 2024-08-19T14:17:35.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-08-19T14:17:35.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:17:36.029 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.7 2024-08-19T14:17:36.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-08-19T14:17:36.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=0aa4eac4-30cb-4c33-b161-da2a12cc721f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.32181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-08-19T14:17:36.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:17:36.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:17:36.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:17:36.060 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:36.060+0000 7ff08860c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:36.062 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:36.062+0000 7ff08860c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:36.063 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:36.063+0000 7ff08860c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:17:36.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-08-19T14:17:36.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:17:36.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-08-19T14:17:36.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:17:36.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:17:36.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:17:36.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:17:36.302 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:17:36.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:17:36.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:17:36.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:17:36.618 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:36.618+0000 7ff08860c640 -1 Falling back to public interface 2024-08-19T14:17:37.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:17:37.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:17:37.570 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:17:37.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:17:37.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:17:37.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:17:37.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:17:38.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:17:38.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:17:38.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:17:38.844 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:17:38.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:17:38.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:17:38.990 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:17:38.990+0000 7ff08860c640 -1 osd.7 0 log_to_monitors true 2024-08-19T14:17:39.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:17:40.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:17:40.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:17:40.216 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:17:40.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:17:40.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:17:40.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:17:40.499 INFO:tasks.workunit.client.0.smithi102.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/2914044436,v1:127.0.0.1:6859/2914044436] [v2:127.0.0.1:6860/2914044436,v1:127.0.0.1:6861/2914044436] exists,up 81618b62-18ef-4bc3-a68e-020802a73019 2024-08-19T14:17:40.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:17:40.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:17:40.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:17:40.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:256: TEST_recovery_scrub_2: create_pool test 32 32 2024-08-19T14:17:40.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 32 32 2024-08-19T14:17:40.947 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' created 2024-08-19T14:17:40.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T14:17:41.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:257: TEST_recovery_scrub_2: wait_for_clean 2024-08-19T14:17:41.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:17:41.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:17:41.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:17:41.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:17:41.962 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:17:41.962 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:17:41.962 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:17:41.962 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:17:41.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:17:42.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:17:42.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:17:42.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:17:42.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:17:42.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:17:42.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:17:42.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:17:42.372 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:17:42.372 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:17:42.372 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:17:42.372 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-08-19T14:17:42.372 INFO:tasks.workunit.client.0.smithi102.stderr:5 2024-08-19T14:17:42.372 INFO:tasks.workunit.client.0.smithi102.stderr:6 2024-08-19T14:17:42.372 INFO:tasks.workunit.client.0.smithi102.stderr:7' 2024-08-19T14:17:42.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:17:42.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:17:42.373 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:17:42.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-08-19T14:17:42.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-08-19T14:17:42.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-08-19T14:17:42.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:17:42.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:17:42.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-08-19T14:17:42.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-08-19T14:17:42.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976' 2024-08-19T14:17:42.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:17:42.841 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:17:43.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-08-19T14:17:43.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-08-19T14:17:43.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453' 2024-08-19T14:17:43.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:17:43.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:17:43.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345931 2024-08-19T14:17:43.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345931 2024-08-19T14:17:43.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931' 2024-08-19T14:17:43.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:17:43.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T14:17:43.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182409 2024-08-19T14:17:43.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182409 2024-08-19T14:17:43.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931 4-107374182409' 2024-08-19T14:17:43.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:17:43.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T14:17:43.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018887 2024-08-19T14:17:43.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018887 2024-08-19T14:17:43.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931 4-107374182409 5-128849018887' 2024-08-19T14:17:43.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:17:43.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T14:17:44.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-08-19T14:17:44.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-08-19T14:17:44.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931 4-107374182409 5-128849018887 6-150323855365' 2024-08-19T14:17:44.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:17:44.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T14:17:44.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-08-19T14:17:44.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-08-19T14:17:44.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931 4-107374182409 5-128849018887 6-150323855365 7-171798691842' 2024-08-19T14:17:44.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:17:44.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-08-19T14:17:44.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:17:44.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:17:44.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-08-19T14:17:44.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:17:44.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-08-19T14:17:44.161 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836498 2024-08-19T14:17:44.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-08-19T14:17:44.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:17:44.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-08-19T14:17:44.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:17:44.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-08-19T14:17:44.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:17:44.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:17:44.448 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-08-19T14:17:44.448 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:17:44.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-08-19T14:17:44.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-08-19T14:17:44.450 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672976 2024-08-19T14:17:44.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:17:44.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672976 2024-08-19T14:17:44.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:17:45.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:17:45.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:17:45.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-08-19T14:17:45.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:17:45.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-08-19T14:17:45.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:17:45.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:17:45.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-08-19T14:17:45.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:17:45.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-08-19T14:17:45.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-08-19T14:17:45.996 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509453 2024-08-19T14:17:45.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:17:46.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509453 2024-08-19T14:17:46.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:17:46.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345931 2024-08-19T14:17:46.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:17:46.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:17:46.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345931 2024-08-19T14:17:46.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:17:46.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345931 2024-08-19T14:17:46.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2024-08-19T14:17:46.269 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345931 2024-08-19T14:17:46.269 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:17:46.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345931 -lt 85899345931 2024-08-19T14:17:46.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:17:46.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182409 2024-08-19T14:17:46.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:17:46.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T14:17:46.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182409 2024-08-19T14:17:46.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:17:46.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182409 2024-08-19T14:17:46.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2024-08-19T14:17:46.546 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182409 2024-08-19T14:17:46.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T14:17:46.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182409 -lt 107374182409 2024-08-19T14:17:46.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:17:46.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018887 2024-08-19T14:17:46.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:17:46.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T14:17:46.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018887 2024-08-19T14:17:46.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:17:46.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018887 2024-08-19T14:17:46.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2024-08-19T14:17:46.813 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.5 seq 128849018887 2024-08-19T14:17:46.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T14:17:47.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018887 2024-08-19T14:17:47.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:17:47.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855365 2024-08-19T14:17:47.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:17:47.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T14:17:47.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855365 2024-08-19T14:17:47.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:17:47.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-08-19T14:17:47.125 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.6 seq 150323855365 2024-08-19T14:17:47.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855365' 2024-08-19T14:17:47.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T14:17:47.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-08-19T14:17:47.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:17:47.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691842 2024-08-19T14:17:47.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:17:47.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T14:17:47.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691842 2024-08-19T14:17:47.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:17:47.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-08-19T14:17:47.395 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.7 seq 171798691842 2024-08-19T14:17:47.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691842' 2024-08-19T14:17:47.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T14:17:47.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691842 2024-08-19T14:17:47.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:17:47.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:17:47.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:17:47.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-08-19T14:17:47.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:17:47.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:17:47.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:17:47.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:17:47.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:17:47.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:17:47.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:17:48.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=25 2024-08-19T14:17:48.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:17:48.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:17:48.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:17:48.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 25 = 32 2024-08-19T14:17:48.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 25 '!=' -1 2024-08-19T14:17:48.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:17:48.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=25 2024-08-19T14:17:48.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:17:48.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:17:48.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:17:48.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:17:48.695 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:17:48.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:17:48.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:17:48.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:17:48.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:17:48.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:17:48.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-08-19T14:17:48.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:17:48.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:17:48.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:17:49.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-08-19T14:17:49.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:17:49.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:17:49.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: ceph osd dump 2024-08-19T14:17:49.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: grep '^pool.*['\'']test['\'']' 2024-08-19T14:17:49.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: awk '{ print $2 }' 2024-08-19T14:17:49.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: poolid=1 2024-08-19T14:17:49.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:260: TEST_recovery_scrub_2: dd if=/dev/urandom of=testdata.32181 bs=1M count=50 2024-08-19T14:17:49.702 INFO:tasks.workunit.client.0.smithi102.stderr:50+0 records in 2024-08-19T14:17:49.702 INFO:tasks.workunit.client.0.smithi102.stderr:50+0 records out 2024-08-19T14:17:49.702 INFO:tasks.workunit.client.0.smithi102.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.142531 s, 368 MB/s 2024-08-19T14:17:49.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: seq 1 40 2024-08-19T14:17:49.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:17:49.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj1 testdata.32181 2024-08-19T14:17:51.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:17:51.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj2 testdata.32181 2024-08-19T14:17:52.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:17:52.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj3 testdata.32181 2024-08-19T14:17:54.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:17:54.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj4 testdata.32181 2024-08-19T14:17:56.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:17:56.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj5 testdata.32181 2024-08-19T14:17:57.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:17:57.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj6 testdata.32181 2024-08-19T14:17:59.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:17:59.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj7 testdata.32181 2024-08-19T14:18:00.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:00.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj8 testdata.32181 2024-08-19T14:18:02.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:02.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj9 testdata.32181 2024-08-19T14:18:03.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:03.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj10 testdata.32181 2024-08-19T14:18:05.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:05.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj11 testdata.32181 2024-08-19T14:18:06.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:06.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj12 testdata.32181 2024-08-19T14:18:08.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:08.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj13 testdata.32181 2024-08-19T14:18:10.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:10.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj14 testdata.32181 2024-08-19T14:18:11.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:11.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj15 testdata.32181 2024-08-19T14:18:13.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:13.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj16 testdata.32181 2024-08-19T14:18:14.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:14.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj17 testdata.32181 2024-08-19T14:18:16.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:16.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj18 testdata.32181 2024-08-19T14:18:17.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:17.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj19 testdata.32181 2024-08-19T14:18:19.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:19.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj20 testdata.32181 2024-08-19T14:18:20.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:20.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj21 testdata.32181 2024-08-19T14:18:22.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:22.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj22 testdata.32181 2024-08-19T14:18:23.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:23.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj23 testdata.32181 2024-08-19T14:18:25.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:25.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj24 testdata.32181 2024-08-19T14:18:27.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:27.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj25 testdata.32181 2024-08-19T14:18:28.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:28.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj26 testdata.32181 2024-08-19T14:18:30.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:30.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj27 testdata.32181 2024-08-19T14:18:31.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:31.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj28 testdata.32181 2024-08-19T14:18:33.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:33.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj29 testdata.32181 2024-08-19T14:18:34.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:34.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj30 testdata.32181 2024-08-19T14:18:36.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:36.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj31 testdata.32181 2024-08-19T14:18:38.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:38.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj32 testdata.32181 2024-08-19T14:18:39.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:39.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj33 testdata.32181 2024-08-19T14:18:41.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:41.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj34 testdata.32181 2024-08-19T14:18:42.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:42.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj35 testdata.32181 2024-08-19T14:18:44.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:44.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj36 testdata.32181 2024-08-19T14:18:45.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:45.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj37 testdata.32181 2024-08-19T14:18:47.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:47.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj38 testdata.32181 2024-08-19T14:18:48.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:48.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj39 testdata.32181 2024-08-19T14:18:50.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-08-19T14:18:50.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj40 testdata.32181 2024-08-19T14:18:51.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: rm -f testdata.32181 2024-08-19T14:18:51.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:267: TEST_recovery_scrub_2: ceph osd pool set test size 3 2024-08-19T14:18:52.103 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 3 2024-08-19T14:18:52.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph pg dump pgs 2024-08-19T14:18:52.368 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T14:18:52.368 INFO:tasks.workunit.client.0.smithi102.stdout:1.1f 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-08-19T14:17:42.181436+0000 49'26 49:58 [5] 5 [5] 5 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.368 INFO:tasks.workunit.client.0.smithi102.stdout:1.1e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-08-19T14:17:42.148388+0000 49'13 49:45 [3] 3 [3] 3 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.368 INFO:tasks.workunit.client.0.smithi102.stdout:1.1d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-08-19T14:17:43.150535+0000 49'13 49:32 [1] 1 [1] 1 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.369 INFO:tasks.workunit.client.0.smithi102.stdout:1.b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-08-19T14:17:43.150324+0000 49'13 49:35 [1] 1 [1] 1 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.369 INFO:tasks.workunit.client.0.smithi102.stdout:1.a 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-08-19T14:17:43.150357+0000 49'13 49:35 [1] 1 [1] 1 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.369 INFO:tasks.workunit.client.0.smithi102.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:17:43.035943+0000 0'0 49:22 [1] 1 [1] 1 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.369 INFO:tasks.workunit.client.0.smithi102.stdout:1.8 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-08-19T14:17:42.181144+0000 49'26 49:49 [5] 5 [5] 5 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.369 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-08-19T14:17:42.545769+0000 49'13 49:37 [6] 6 [6] 6 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.369 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 1 0 0 0 0 12582912 0 0 3 0 3 active+clean 2024-08-19T14:17:42.675779+0000 49'3 49:27 [6] 6 [6] 6 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.369 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-08-19T14:17:42.675777+0000 49'39 49:63 [4] 4 [4] 4 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.369 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-08-19T14:17:42.058514+0000 49'52 49:77 [3] 3 [3] 3 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.369 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-08-19T14:17:42.675776+0000 49'52 49:77 [4] 4 [4] 4 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.369 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-08-19T14:17:41.795767+0000 49'13 49:41 [0] 0 [0] 0 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.369 INFO:tasks.workunit.client.0.smithi102.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:17:42.148419+0000 0'0 49:23 [3] 3 [3] 3 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.370 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-08-19T14:17:42.058471+0000 49'13 49:45 [5] 5 [5] 5 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.370 INFO:tasks.workunit.client.0.smithi102.stdout:1.1c 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-08-19T14:17:41.796409+0000 49'13 49:34 [0] 0 [0] 0 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.370 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-08-19T14:17:43.402188+0000 49'13 49:41 [7] 7 [7] 7 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.370 INFO:tasks.workunit.client.0.smithi102.stdout:1.d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-08-19T14:17:43.402245+0000 49'13 49:31 [7] 7 [7] 7 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.370 INFO:tasks.workunit.client.0.smithi102.stdout:1.e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-08-19T14:17:43.150426+0000 49'13 49:34 [1] 1 [1] 1 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.370 INFO:tasks.workunit.client.0.smithi102.stdout:1.f 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-08-19T14:17:42.675781+0000 49'13 49:34 [6] 6 [6] 6 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.370 INFO:tasks.workunit.client.0.smithi102.stdout:1.10 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-08-19T14:17:42.181289+0000 49'13 49:35 [5] 5 [5] 5 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.370 INFO:tasks.workunit.client.0.smithi102.stdout:1.11 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-08-19T14:17:41.796133+0000 49'13 49:35 [0] 0 [0] 0 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.370 INFO:tasks.workunit.client.0.smithi102.stdout:1.12 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-08-19T14:17:41.795687+0000 49'26 49:47 [0] 0 [0] 0 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.370 INFO:tasks.workunit.client.0.smithi102.stdout:1.13 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-08-19T14:17:42.545807+0000 49'13 49:33 [4] 4 [4] 4 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.370 INFO:tasks.workunit.client.0.smithi102.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:17:42.058408+0000 0'0 49:21 [2] 2 [2] 2 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.371 INFO:tasks.workunit.client.0.smithi102.stdout:1.15 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-08-19T14:17:43.150797+0000 49'39 49:59 [1] 1 [1] 1 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.371 INFO:tasks.workunit.client.0.smithi102.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:17:42.675781+0000 0'0 49:20 [6] 6 [6] 6 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.371 INFO:tasks.workunit.client.0.smithi102.stdout:1.17 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-08-19T14:17:41.796297+0000 49'13 49:34 [0] 0 [0] 0 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.371 INFO:tasks.workunit.client.0.smithi102.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:17:42.181075+0000 0'0 49:21 [5] 5 [5] 5 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.371 INFO:tasks.workunit.client.0.smithi102.stdout:1.19 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-08-19T14:17:42.058430+0000 49'13 49:34 [3] 3 [3] 3 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.371 INFO:tasks.workunit.client.0.smithi102.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:17:43.150885+0000 0'0 49:20 [1] 1 [1] 1 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.371 INFO:tasks.workunit.client.0.smithi102.stdout:1.1b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-08-19T14:17:41.796293+0000 49'13 49:34 [0] 0 [0] 0 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:18:52.371 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:18:52.371 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T14:18:52.372 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:18:52.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:275: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active 2024-08-19T14:18:52.573 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T14:18:52.573 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_recovery_max_active": "0" 2024-08-19T14:18:52.573 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:18:52.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:277: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_hdd 2024-08-19T14:18:52.841 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T14:18:52.841 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_recovery_max_active_hdd": "3" 2024-08-19T14:18:52.841 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:18:52.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:278: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_ssd 2024-08-19T14:18:53.136 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T14:18:53.136 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_recovery_max_active_ssd": "10" 2024-08-19T14:18:53.136 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:18:53.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:281: TEST_recovery_scrub_2: count=0 2024-08-19T14:18:53.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-08-19T14:18:53.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-08-19T14:18:53.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-08-19T14:18:53.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-08-19T14:18:53.146 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-08-19T14:18:53.581 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:18:53.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-08-19T14:18:53.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-08-19T14:18:55.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 0 -eq 10 2024-08-19T14:18:55.592 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 0 + 1 2024-08-19T14:18:55.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=1 2024-08-19T14:18:55.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-08-19T14:18:55.595 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-08-19T14:18:55.595 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-08-19T14:18:55.595 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-08-19T14:18:55.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-08-19T14:18:55.858 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:18:55.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-08-19T14:18:55.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-08-19T14:18:57.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 1 -eq 10 2024-08-19T14:18:57.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 1 + 1 2024-08-19T14:18:57.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=2 2024-08-19T14:18:57.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-08-19T14:18:57.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-08-19T14:18:57.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-08-19T14:18:57.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-08-19T14:18:57.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-08-19T14:18:58.132 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:18:58.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-08-19T14:18:58.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-08-19T14:19:00.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 2 -eq 10 2024-08-19T14:19:00.146 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 2 + 1 2024-08-19T14:19:00.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=3 2024-08-19T14:19:00.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-08-19T14:19:00.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-08-19T14:19:00.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-08-19T14:19:00.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-08-19T14:19:00.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-08-19T14:19:00.419 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:19:00.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-08-19T14:19:00.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-08-19T14:19:02.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 3 -eq 10 2024-08-19T14:19:02.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 3 + 1 2024-08-19T14:19:02.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=4 2024-08-19T14:19:02.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-08-19T14:19:02.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-08-19T14:19:02.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-08-19T14:19:02.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-08-19T14:19:02.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-08-19T14:19:02.697 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:19:02.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 2 -ge 2 2024-08-19T14:19:02.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:288: TEST_recovery_scrub_2: break 2024-08-19T14:19:02.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:298: TEST_recovery_scrub_2: ceph pg dump pgs 2024-08-19T14:19:02.963 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T14:19:02.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.1f 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-08-19T14:18:54.255827+0000 49'26 52:70 [5,0,2] 5 [5,0,2] 5 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.1e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-08-19T14:18:54.457073+0000 49'13 52:56 [3,0,7] 3 [3,0,7] 3 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.1d 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-08-19T14:18:53.870568+0000 49'26 52:56 [1,3,0] 1 [1,3,0] 1 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-08-19T14:18:54.062875+0000 49'13 52:46 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.a 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-08-19T14:18:53.197144+0000 49'13 52:47 [1,6,2] 1 [1,6,2] 1 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:18:53.870368+0000 0'0 52:35 [1,4,5] 1 [1,4,5] 1 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:19:02.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.8 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-08-19T14:18:54.616229+0000 49'26 52:59 [5,4,3] 5 [5,4,3] 5 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-08-19T14:18:54.255367+0000 49'13 52:47 [6,7,3] 6 [6,7,3] 6 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-08-19T14:18:54.063163+0000 49'13 52:48 [6,7,3] 6 [6,7,3] 6 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-08-19T14:18:54.162861+0000 49'39 52:73 [4,2,6] 4 [4,2,6] 4 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-08-19T14:18:54.457057+0000 49'52 52:87 [3,0,6] 3 [3,0,6] 3 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-08-19T14:18:54.616236+0000 49'52 52:87 [4,1,2] 4 [4,1,2] 4 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-08-19T14:18:54.616208+0000 49'13 52:50 [0,7,1] 0 [0,7,1] 0 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:18:54.256393+0000 0'0 52:35 [3,0,4] 3 [3,0,4] 3 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:19:02.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-08-19T14:18:54.457228+0000 49'13 52:57 [5,0,4] 5 [5,0,4] 5 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.1c 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-08-19T14:18:54.257444+0000 49'13 52:42 [0,3,4] 0 [0,3,4] 0 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-08-19T14:18:54.256515+0000 49'13 52:51 [7,0,2] 7 [7,0,2] 7 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.d 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-08-19T14:18:54.255246+0000 49'13 52:40 [7,4,2] 7 [7,4,2] 7 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-08-19T14:18:53.565422+0000 49'13 52:44 [1,4,2] 1 [1,4,2] 1 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.f 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-08-19T14:18:54.255326+0000 49'13 52:43 [6,0,3] 6 [6,0,3] 6 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.10 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-08-19T14:18:54.255936+0000 49'13 52:45 [5,7,6] 5 [5,7,6] 5 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.966 INFO:tasks.workunit.client.0.smithi102.stdout:1.11 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-08-19T14:18:54.456572+0000 49'13 52:44 [0,1,3] 0 [0,1,3] 0 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.966 INFO:tasks.workunit.client.0.smithi102.stdout:1.12 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-08-19T14:18:54.456947+0000 49'26 52:56 [0,1,6] 0 [0,1,6] 0 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.966 INFO:tasks.workunit.client.0.smithi102.stdout:1.13 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-08-19T14:18:54.256544+0000 49'13 52:43 [4,6,3] 4 [4,6,3] 4 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.966 INFO:tasks.workunit.client.0.smithi102.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:18:54.255912+0000 0'0 52:33 [2,0,4] 2 [2,0,4] 2 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:19:02.966 INFO:tasks.workunit.client.0.smithi102.stdout:1.15 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-08-19T14:18:53.237929+0000 49'39 52:70 [1,5,3] 1 [1,5,3] 1 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.966 INFO:tasks.workunit.client.0.smithi102.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:18:54.062345+0000 0'0 52:32 [6,0,7] 6 [6,0,7] 6 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:19:02.966 INFO:tasks.workunit.client.0.smithi102.stdout:1.17 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-08-19T14:18:54.457126+0000 49'13 52:42 [0,4,5] 0 [0,4,5] 0 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.966 INFO:tasks.workunit.client.0.smithi102.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:18:54.255325+0000 0'0 52:32 [5,2,3] 5 [5,2,3] 5 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:19:02.966 INFO:tasks.workunit.client.0.smithi102.stdout:1.19 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-08-19T14:18:54.616308+0000 49'13 52:43 [3,0,4] 3 [3,0,4] 3 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.966 INFO:tasks.workunit.client.0.smithi102.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-19T14:18:53.869608+0000 0'0 52:32 [1,7,4] 1 [1,7,4] 1 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 periodic scrub scheduled @ 2024-08-20T14:17:40.848075+0000 0 0 2024-08-19T14:19:02.966 INFO:tasks.workunit.client.0.smithi102.stdout:1.1b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-08-19T14:18:54.254969+0000 49'13 52:42 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-19T14:17:40.848075+0000 0'0 2024-08-19T14:17:40.848075+0000 0 0 no scrub is scheduled 0 0 2024-08-19T14:19:02.966 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:19:02.967 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T14:19:02.967 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:19:02.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: pids= 2024-08-19T14:19:02.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:301: TEST_recovery_scrub_2: recov_scrub_count=0 2024-08-19T14:19:02.975 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: expr 32 - 1 2024-08-19T14:19:02.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: seq 0 31 2024-08-19T14:19:02.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:02.979 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 0 2024-08-19T14:19:02.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.0 2024-08-19T14:19:02.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:02.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:02.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53744"' 2024-08-19T14:19:02.979 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53744' 2024-08-19T14:19:02.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:02.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.0 2024-08-19T14:19:02.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 1 2024-08-19T14:19:02.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53747: /' 2024-08-19T14:19:02.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1 2024-08-19T14:19:02.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:02.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:02.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53749"' 2024-08-19T14:19:02.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53749' 2024-08-19T14:19:02.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:02.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1 2024-08-19T14:19:02.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 2 2024-08-19T14:19:02.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53754: /' 2024-08-19T14:19:02.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.2 2024-08-19T14:19:02.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:02.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:02.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53756"' 2024-08-19T14:19:02.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53756' 2024-08-19T14:19:02.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:02.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.2 2024-08-19T14:19:02.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53761: /' 2024-08-19T14:19:02.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 3 2024-08-19T14:19:02.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.3 2024-08-19T14:19:02.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:02.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:02.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53764"' 2024-08-19T14:19:02.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53764' 2024-08-19T14:19:02.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:02.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 4 2024-08-19T14:19:02.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.4 2024-08-19T14:19:02.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.3 2024-08-19T14:19:02.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:02.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:02.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53768: /' 2024-08-19T14:19:02.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53769"' 2024-08-19T14:19:02.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53769' 2024-08-19T14:19:02.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:02.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 5 2024-08-19T14:19:02.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.5 2024-08-19T14:19:02.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:02.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:02.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53773"' 2024-08-19T14:19:02.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.4 2024-08-19T14:19:02.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53773' 2024-08-19T14:19:02.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53774: /' 2024-08-19T14:19:02.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:02.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.5 2024-08-19T14:19:02.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53777: /' 2024-08-19T14:19:02.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 6 2024-08-19T14:19:02.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.6 2024-08-19T14:19:02.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:02.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:02.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53782"' 2024-08-19T14:19:02.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53782' 2024-08-19T14:19:02.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:02.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53785: /' 2024-08-19T14:19:02.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 7 2024-08-19T14:19:02.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.7 2024-08-19T14:19:02.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:02.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:02.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53786"' 2024-08-19T14:19:02.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53786' 2024-08-19T14:19:02.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:02.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.7 2024-08-19T14:19:02.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53793: /' 2024-08-19T14:19:02.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.6 2024-08-19T14:19:03.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 8 2024-08-19T14:19:03.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.8 2024-08-19T14:19:03.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53800"' 2024-08-19T14:19:03.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53800' 2024-08-19T14:19:03.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 9 2024-08-19T14:19:03.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.9 2024-08-19T14:19:03.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53802"' 2024-08-19T14:19:03.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53802' 2024-08-19T14:19:03.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 10 2024-08-19T14:19:03.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53805: /' 2024-08-19T14:19:03.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.a 2024-08-19T14:19:03.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.8 2024-08-19T14:19:03.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53806"' 2024-08-19T14:19:03.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53806' 2024-08-19T14:19:03.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53810: /' 2024-08-19T14:19:03.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.9 2024-08-19T14:19:03.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 11 2024-08-19T14:19:03.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.b 2024-08-19T14:19:03.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53814"' 2024-08-19T14:19:03.044 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53814' 2024-08-19T14:19:03.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 12 2024-08-19T14:19:03.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53812: /' 2024-08-19T14:19:03.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.a 2024-08-19T14:19:03.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.c 2024-08-19T14:19:03.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53821"' 2024-08-19T14:19:03.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53821' 2024-08-19T14:19:03.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53820: /' 2024-08-19T14:19:03.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 13 2024-08-19T14:19:03.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.d 2024-08-19T14:19:03.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53827"' 2024-08-19T14:19:03.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53827' 2024-08-19T14:19:03.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.b 2024-08-19T14:19:03.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 14 2024-08-19T14:19:03.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.c 2024-08-19T14:19:03.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53831: /' 2024-08-19T14:19:03.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.e 2024-08-19T14:19:03.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53835"' 2024-08-19T14:19:03.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53835' 2024-08-19T14:19:03.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 15 2024-08-19T14:19:03.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53838: /' 2024-08-19T14:19:03.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.f 2024-08-19T14:19:03.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53839"' 2024-08-19T14:19:03.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53839' 2024-08-19T14:19:03.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.d 2024-08-19T14:19:03.091 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 16 2024-08-19T14:19:03.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.10 2024-08-19T14:19:03.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53846"' 2024-08-19T14:19:03.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53846' 2024-08-19T14:19:03.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.e 2024-08-19T14:19:03.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53845: /' 2024-08-19T14:19:03.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 17 2024-08-19T14:19:03.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.11 2024-08-19T14:19:03.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53856"' 2024-08-19T14:19:03.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53856' 2024-08-19T14:19:03.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53849: /' 2024-08-19T14:19:03.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.f 2024-08-19T14:19:03.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.10 2024-08-19T14:19:03.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 18 2024-08-19T14:19:03.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53859: /' 2024-08-19T14:19:03.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.12 2024-08-19T14:19:03.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53868"' 2024-08-19T14:19:03.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53868' 2024-08-19T14:19:03.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 19 2024-08-19T14:19:03.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.13 2024-08-19T14:19:03.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53871"' 2024-08-19T14:19:03.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53871' 2024-08-19T14:19:03.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.12 2024-08-19T14:19:03.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53878: /' 2024-08-19T14:19:03.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.11 2024-08-19T14:19:03.136 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 20 2024-08-19T14:19:03.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.14 2024-08-19T14:19:03.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53887"' 2024-08-19T14:19:03.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53887' 2024-08-19T14:19:03.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 21 2024-08-19T14:19:03.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53893: /' 2024-08-19T14:19:03.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.13 2024-08-19T14:19:03.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.15 2024-08-19T14:19:03.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53899"' 2024-08-19T14:19:03.146 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53899' 2024-08-19T14:19:03.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53874: /' 2024-08-19T14:19:03.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 22 2024-08-19T14:19:03.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53901: /' 2024-08-19T14:19:03.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.14 2024-08-19T14:19:03.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.16 2024-08-19T14:19:03.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53914"' 2024-08-19T14:19:03.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53914' 2024-08-19T14:19:03.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.15 2024-08-19T14:19:03.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53906: /' 2024-08-19T14:19:03.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 23 2024-08-19T14:19:03.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.17 2024-08-19T14:19:03.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53927"' 2024-08-19T14:19:03.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53927' 2024-08-19T14:19:03.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.179 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 24 2024-08-19T14:19:03.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.18 2024-08-19T14:19:03.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53925: /' 2024-08-19T14:19:03.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53934"' 2024-08-19T14:19:03.180 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53934' 2024-08-19T14:19:03.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.17 2024-08-19T14:19:03.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.16 2024-08-19T14:19:03.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 25 2024-08-19T14:19:03.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.19 2024-08-19T14:19:03.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53948"' 2024-08-19T14:19:03.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53948' 2024-08-19T14:19:03.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53933: /' 2024-08-19T14:19:03.197 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 26 2024-08-19T14:19:03.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53956: /' 2024-08-19T14:19:03.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1a 2024-08-19T14:19:03.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 53979"' 2024-08-19T14:19:03.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 53979' 2024-08-19T14:19:03.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.18 2024-08-19T14:19:03.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/53961: /' 2024-08-19T14:19:03.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.19 2024-08-19T14:19:03.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 27 2024-08-19T14:19:03.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1a 2024-08-19T14:19:03.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1b 2024-08-19T14:19:03.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 54004"' 2024-08-19T14:19:03.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 54004' 2024-08-19T14:19:03.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 28 2024-08-19T14:19:03.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1c 2024-08-19T14:19:03.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 54020"' 2024-08-19T14:19:03.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 54020' 2024-08-19T14:19:03.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/54001: /' 2024-08-19T14:19:03.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/54007: /' 2024-08-19T14:19:03.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1b 2024-08-19T14:19:03.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 29 2024-08-19T14:19:03.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1d 2024-08-19T14:19:03.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 54027"' 2024-08-19T14:19:03.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 54027' 2024-08-19T14:19:03.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/54024: /' 2024-08-19T14:19:03.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1c 2024-08-19T14:19:03.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 30 2024-08-19T14:19:03.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1e 2024-08-19T14:19:03.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 54065"' 2024-08-19T14:19:03.281 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 54065' 2024-08-19T14:19:03.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-08-19T14:19:03.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 31 2024-08-19T14:19:03.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1f 2024-08-19T14:19:03.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:19:03.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:19:03.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 54072"' 2024-08-19T14:19:03.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 54072' 2024-08-19T14:19:03.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:306: TEST_recovery_scrub_2: wait_background_check pids 2024-08-19T14:19:03.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: pids=' 53744 53749 53756 53764 53769 53773 53782 53786 53800 53802 53806 53814 53821 53827 53835 53839 53846 53856 53868 53871 53887 53899 53914 53927 53934 53948 53979 54004 54020 54027 54065 54072' 2024-08-19T14:19:03.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: return_code=0 2024-08-19T14:19:03.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:03.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53744 2024-08-19T14:19:03.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/54074: /' 2024-08-19T14:19:03.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/54079: /' 2024-08-19T14:19:03.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1e 2024-08-19T14:19:03.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/54100: /' 2024-08-19T14:19:03.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1f 2024-08-19T14:19:03.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1d 2024-08-19T14:19:09.233 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.a 2024-08-19T14:19:09.234 INFO:tasks.workunit.client.0.smithi102.stderr:53812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.a 2024-08-19T14:19:09.234 INFO:tasks.workunit.client.0.smithi102.stderr:53812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-08-19T14:19:09.234 INFO:tasks.workunit.client.0.smithi102.stderr:53812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:09.234 INFO:tasks.workunit.client.0.smithi102.stderr:53812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:09.234 INFO:tasks.workunit.client.0.smithi102.stderr:53812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-08-19T14:19:09.234 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:09.234 INFO:tasks.workunit.client.0.smithi102.stderr:53812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:09.234 INFO:tasks.workunit.client.0.smithi102.stderr:53812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.a query 2024-08-19T14:19:09.234 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:09.234 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:09.234 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:09.235 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.a 2024-08-19T14:19:09.235 INFO:tasks.workunit.client.0.smithi102.stderr:53812: instructing pg 1.a on osd.1 to scrub 2024-08-19T14:19:09.235 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.a") | .state' 2024-08-19T14:19:09.235 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:09.235 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:09.235 INFO:tasks.workunit.client.0.smithi102.stderr:53812: dumped pgs 2024-08-19T14:19:09.235 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-08-19T14:19:09.235 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.a 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:09.235 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.a 2024-08-19T14:19:09.235 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:09.235 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:09.235 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:09.235 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:09.236 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:09.236 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:09.236 INFO:tasks.workunit.client.0.smithi102.stderr:53812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.a last_scrub_stamp 2024-08-19T14:19:09.236 INFO:tasks.workunit.client.0.smithi102.stderr:53812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-08-19T14:19:09.236 INFO:tasks.workunit.client.0.smithi102.stderr:53812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:09.236 INFO:tasks.workunit.client.0.smithi102.stderr:53812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:09.668 INFO:tasks.workunit.client.0.smithi102.stderr:53812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | se53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.12 2024-08-19T14:19:09.668 INFO:tasks.workunit.client.0.smithi102.stderr:53874: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.12 2024-08-19T14:19:09.668 INFO:tasks.workunit.client.0.smithi102.stderr:53874: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-08-19T14:19:09.669 INFO:tasks.workunit.client.0.smithi102.stderr:53874: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:09.669 INFO:tasks.workunit.client.0.smithi102.stderr:53874: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-08-19T14:19:09.669 INFO:tasks.workunit.client.0.smithi102.stderr:53874: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:09.669 INFO:tasks.workunit.client.0.smithi102.stderr:53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:09.669 INFO:tasks.workunit.client.0.smithi102.stderr:53874: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:09.669 INFO:tasks.workunit.client.0.smithi102.stderr:53874: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.12 query 2024-08-19T14:19:09.669 INFO:tasks.workunit.client.0.smithi102.stderr:53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:09.669 INFO:tasks.workunit.client.0.smithi102.stderr:53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:09.669 INFO:tasks.workunit.client.0.smithi102.stderr:53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:09.669 INFO:tasks.workunit.client.0.smithi102.stderr:53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.12 2024-08-19T14:19:09.669 INFO:tasks.workunit.client.0.smithi102.stderr:53874: instructing pg 1.12 on osd.0 to scrub 2024-08-19T14:19:09.669 INFO:tasks.workunit.client.0.smithi102.stderr:53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:09.670 INFO:tasks.workunit.client.0.smithi102.stderr:53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:09.670 INFO:tasks.workunit.client.0.smithi102.stderr:53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.12") | .state' 2024-08-19T14:19:09.670 INFO:tasks.workunit.client.0.smithi102.stderr:53874: dumped pgs 2024-08-19T14:19:09.670 INFO:tasks.workunit.client.0.smithi102.stderr:53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.12 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:09.670 INFO:tasks.workunit.client.0.smithi102.stderr:53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.12 2024-08-19T14:19:09.670 INFO:tasks.workunit.client.0.smithi102.stderr:53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:09.670 INFO:tasks.workunit.client.0.smithi102.stderr:53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:09.670 INFO:tasks.workunit.client.0.smithi102.stderr:53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:09.670 INFO:tasks.workunit.client.0.smithi102.stderr:53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:09.670 INFO:tasks.workunit.client.0.smithi102.stderr:53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:53874: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.12 last_scrub_stamp 2024-08-19T14:19:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:53874: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-08-19T14:19:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:53874: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:53874: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:53874: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-08-19T14:19:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:53874: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:09.740 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1 2024-08-19T14:19:09.740 INFO:tasks.workunit.client.0.smithi102.stderr:53754: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1 2024-08-19T14:19:09.740 INFO:tasks.workunit.client.0.smithi102.stderr:53754: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-08-19T14:19:09.740 INFO:tasks.workunit.client.0.smithi102.stderr:53754: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:09.741 INFO:tasks.workunit.client.0.smithi102.stderr:53754: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:09.741 INFO:tasks.workunit.client.0.smithi102.stderr:53754: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-08-19T14:19:09.741 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:09.741 INFO:tasks.workunit.client.0.smithi102.stderr:53754: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:09.741 INFO:tasks.workunit.client.0.smithi102.stderr:53754: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1 query 2024-08-19T14:19:09.741 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:09.741 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:09.741 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:09.741 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1 2024-08-19T14:19:09.741 INFO:tasks.workunit.client.0.smithi102.stderr:53754: instructing pg 1.1 on osd.5 to scrub 2024-08-19T14:19:09.741 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:09.742 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1") | .state' 2024-08-19T14:19:09.742 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:09.742 INFO:tasks.workunit.client.0.smithi102.stderr:53754: dumped pgs 2024-08-19T14:19:09.742 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:09.742 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1 2024-08-19T14:19:09.742 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:09.742 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:09.742 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:09.742 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:09.742 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:09.742 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:09.742 INFO:tasks.workunit.client.0.smithi102.stderr:53754: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-08-19T14:19:09.742 INFO:tasks.workunit.client.0.smithi102.stderr:53754: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-08-19T14:19:09.743 INFO:tasks.workunit.client.0.smithi102.stderr:53754: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:09.743 INFO:tasks.workunit.client.0.smithi102.stderr:53754: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:09.743 INFO:tasks.workunit.client.0.smithi102.stderr:53754: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-08-19T14:19:09.743 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:09.743 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:09.743 INFO:tasks.workunit.client.0.smithi102.stderr:53754: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:09.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:09.885 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.6 2024-08-19T14:19:09.885 INFO:tasks.workunit.client.0.smithi102.stderr:53785: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.6 2024-08-19T14:19:09.886 INFO:tasks.workunit.client.0.smithi102.stderr:53785: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-08-19T14:19:09.886 INFO:tasks.workunit.client.0.smithi102.stderr:53785: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:09.886 INFO:tasks.workunit.client.0.smithi102.stderr:53785: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-08-19T14:19:09.886 INFO:tasks.workunit.client.0.smithi102.stderr:53785: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:09.886 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:09.886 INFO:tasks.workunit.client.0.smithi102.stderr:53785: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.6 query 2024-08-19T14:19:09.886 INFO:tasks.workunit.client.0.smithi102.stderr:53785: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:09.886 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:09.886 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:09.886 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:09.886 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.6 2024-08-19T14:19:09.886 INFO:tasks.workunit.client.0.smithi102.stderr:53785: instructing pg 1.6 on osd.6 to scrub 2024-08-19T14:19:09.887 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:09.887 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.6") | .state' 2024-08-19T14:19:09.887 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:09.887 INFO:tasks.workunit.client.0.smithi102.stderr:53785: dumped pgs 2024-08-19T14:19:09.887 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-08-19T14:19:09.887 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.6 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:09.887 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.6 2024-08-19T14:19:09.887 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:09.887 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:09.887 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:09.887 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:09.887 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:09.887 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:09.888 INFO:tasks.workunit.client.0.smithi102.stderr:53785: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-08-19T14:19:09.888 INFO:tasks.workunit.client.0.smithi102.stderr:53785: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-08-19T14:19:09.888 INFO:tasks.workunit.client.0.smithi102.stderr:53785: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:09.888 INFO:tasks.workunit.client.0.smithi102.stderr:53785: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-08-19T14:19:10.348 INFO:tasks.workunit.client.0.smithi102.stderr:53785: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.5 2024-08-19T14:19:10.348 INFO:tasks.workunit.client.0.smithi102.stderr:53777: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.5 2024-08-19T14:19:10.348 INFO:tasks.workunit.client.0.smithi102.stderr:53777: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-08-19T14:19:10.348 INFO:tasks.workunit.client.0.smithi102.stderr:53777: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:10.348 INFO:tasks.workunit.client.0.smithi102.stderr:53777: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:10.348 INFO:tasks.workunit.client.0.smithi102.stderr:53777: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-08-19T14:19:10.348 INFO:tasks.workunit.client.0.smithi102.stderr:53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:10.348 INFO:tasks.workunit.client.0.smithi102.stderr:53777: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.5 query 2024-08-19T14:19:10.348 INFO:tasks.workunit.client.0.smithi102.stderr:53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:10.348 INFO:tasks.workunit.client.0.smithi102.stderr:53777: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:10.349 INFO:tasks.workunit.client.0.smithi102.stderr:53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:10.349 INFO:tasks.workunit.client.0.smithi102.stderr:53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:10.349 INFO:tasks.workunit.client.0.smithi102.stderr:53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.5 2024-08-19T14:19:10.349 INFO:tasks.workunit.client.0.smithi102.stderr:53777: instructing pg 1.5 on osd.4 to scrub 2024-08-19T14:19:10.349 INFO:tasks.workunit.client.0.smithi102.stderr:53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.5") | .state' 2024-08-19T14:19:10.349 INFO:tasks.workunit.client.0.smithi102.stderr:53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:10.349 INFO:tasks.workunit.client.0.smithi102.stderr:53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:10.349 INFO:tasks.workunit.client.0.smithi102.stderr:53777: dumped pgs 2024-08-19T14:19:10.349 INFO:tasks.workunit.client.0.smithi102.stderr:53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.5 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:10.349 INFO:tasks.workunit.client.0.smithi102.stderr:53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.5 2024-08-19T14:19:10.349 INFO:tasks.workunit.client.0.smithi102.stderr:53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:10.349 INFO:tasks.workunit.client.0.smithi102.stderr:53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:10.349 INFO:tasks.workunit.client.0.smithi102.stderr:53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:53777: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-08-19T14:19:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:53777: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-08-19T14:19:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:53777: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:53777: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:53777: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-08-19T14:19:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:53777: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:10.673 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.3 2024-08-19T14:19:10.674 INFO:tasks.workunit.client.0.smithi102.stderr:53768: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.3 2024-08-19T14:19:10.674 INFO:tasks.workunit.client.0.smithi102.stderr:53768: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T14:19:10.674 INFO:tasks.workunit.client.0.smithi102.stderr:53768: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:10.674 INFO:tasks.workunit.client.0.smithi102.stderr:53768: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:10.674 INFO:tasks.workunit.client.0.smithi102.stderr:53768: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T14:19:10.674 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:10.674 INFO:tasks.workunit.client.0.smithi102.stderr:53768: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:10.674 INFO:tasks.workunit.client.0.smithi102.stderr:53768: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.3 query 2024-08-19T14:19:10.674 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:10.674 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:10.674 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:10.674 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.3 2024-08-19T14:19:10.675 INFO:tasks.workunit.client.0.smithi102.stderr:53768: instructing pg 1.3 on osd.4 to scrub 2024-08-19T14:19:10.675 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:10.675 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.3") | .state' 2024-08-19T14:19:10.675 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:10.675 INFO:tasks.workunit.client.0.smithi102.stderr:53768: dumped pgs 2024-08-19T14:19:10.675 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.3 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:10.675 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.3 2024-08-19T14:19:10.675 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:10.675 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:10.675 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:10.675 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:10.675 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:10.675 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:10.676 INFO:tasks.workunit.client.0.smithi102.stderr:53768: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T14:19:10.676 INFO:tasks.workunit.client.0.smithi102.stderr:53768: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T14:19:10.676 INFO:tasks.workunit.client.0.smithi102.stderr:53768: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:10.676 INFO:tasks.workunit.client.0.smithi102.stderr:53768: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T14:19:10.676 INFO:tasks.workunit.client.0.smithi102.stderr:53768: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:10.676 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:10.676 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:10.676 INFO:tasks.workunit.client.0.smithi102.stderr:53768: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:10.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:10.879 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.2 2024-08-19T14:19:10.879 INFO:tasks.workunit.client.0.smithi102.stderr:53761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.2 2024-08-19T14:19:10.879 INFO:tasks.workunit.client.0.smithi102.stderr:53761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-08-19T14:19:10.879 INFO:tasks.workunit.client.0.smithi102.stderr:53761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:10.879 INFO:tasks.workunit.client.0.smithi102.stderr:53761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:10.880 INFO:tasks.workunit.client.0.smithi102.stderr:53761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-08-19T14:19:10.880 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:10.880 INFO:tasks.workunit.client.0.smithi102.stderr:53761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:10.880 INFO:tasks.workunit.client.0.smithi102.stderr:53761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.2 query 2024-08-19T14:19:10.880 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:10.880 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:10.880 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:10.880 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.2 2024-08-19T14:19:10.880 INFO:tasks.workunit.client.0.smithi102.stderr:53761: instructing pg 1.2 on osd.0 to scrub 2024-08-19T14:19:10.880 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.2") | .state' 2024-08-19T14:19:10.880 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:10.880 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:10.880 INFO:tasks.workunit.client.0.smithi102.stderr:53761: dumped pgs 2024-08-19T14:19:10.881 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.2 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:10.881 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.2 2024-08-19T14:19:10.881 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:10.881 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:10.881 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:10.881 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:10.881 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:10.881 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:10.881 INFO:tasks.workunit.client.0.smithi102.stderr:53761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-08-19T14:19:10.881 INFO:tasks.workunit.client.0.smithi102.stderr:53761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-08-19T14:19:10.881 INFO:tasks.workunit.client.0.smithi102.stderr:53761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:10.881 INFO:tasks.workunit.client.0.smithi102.stderr:53761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:10.881 INFO:tasks.workunit.client.0.smithi102.stderr:53761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-08-19T14:19:10.882 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:10.882 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:10.882 INFO:tasks.workunit.client.0.smithi102.stderr:53761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:10.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:10.975 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1f 2024-08-19T14:19:10.976 INFO:tasks.workunit.client.0.smithi102.stderr:54100: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1f 2024-08-19T14:19:10.976 INFO:tasks.workunit.client.0.smithi102.stderr:54100: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-08-19T14:19:10.976 INFO:tasks.workunit.client.0.smithi102.stderr:54100: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:10.976 INFO:tasks.workunit.client.0.smithi102.stderr:54100: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-08-19T14:19:10.976 INFO:tasks.workunit.client.0.smithi102.stderr:54100: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:10.976 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:10.976 INFO:tasks.workunit.client.0.smithi102.stderr:54100: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1f query 2024-08-19T14:19:10.976 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:10.976 INFO:tasks.workunit.client.0.smithi102.stderr:54100: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:10.976 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:10.976 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:10.976 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1f 2024-08-19T14:19:10.976 INFO:tasks.workunit.client.0.smithi102.stderr:54100: instructing pg 1.1f on osd.5 to scrub 2024-08-19T14:19:10.977 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1f") | .state' 2024-08-19T14:19:10.977 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:10.977 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:10.977 INFO:tasks.workunit.client.0.smithi102.stderr:54100: dumped pgs 2024-08-19T14:19:10.977 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1f 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:10.977 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1f 2024-08-19T14:19:10.977 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:10.977 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:10.977 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:10.977 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:10.977 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:10.977 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:10.977 INFO:tasks.workunit.client.0.smithi102.stderr:54100: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1f last_scrub_stamp 2024-08-19T14:19:10.978 INFO:tasks.workunit.client.0.smithi102.stderr:54100: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-08-19T14:19:10.978 INFO:tasks.workunit.client.0.smithi102.stderr:54100: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:10.978 INFO:tasks.workunit.client.0.smithi102.stderr:54100: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:10.978 INFO:tasks.workunit.client.0.smithi102.stderr:54100: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-08-19T14:19:10.978 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:10.978 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:10.978 INFO:tasks.workunit.client.0.smithi102.stderr:54100: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:10.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:11.163 INFO:tasks.workunit.client.0.smithi102.stderr:lect(.pgid=="1.a") | .last_scrub_stamp' 2024-08-19T14:19:11.163 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:11.163 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:11.163 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-08-19T14:19:11.163 INFO:tasks.workunit.client.0.smithi102.stderr:53812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-08-19T14:19:11.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-08-19T14:19:11.230 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1b 2024-08-19T14:19:11.230 INFO:tasks.workunit.client.0.smithi102.stderr:54007: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1b 2024-08-19T14:19:11.230 INFO:tasks.workunit.client.0.smithi102.stderr:54007: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-08-19T14:19:11.230 INFO:tasks.workunit.client.0.smithi102.stderr:54007: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.231 INFO:tasks.workunit.client.0.smithi102.stderr:54007: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-08-19T14:19:11.231 INFO:tasks.workunit.client.0.smithi102.stderr:54007: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.231 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.231 INFO:tasks.workunit.client.0.smithi102.stderr:54007: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:11.231 INFO:tasks.workunit.client.0.smithi102.stderr:54007: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1b query 2024-08-19T14:19:11.231 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:11.231 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:11.231 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:11.231 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1b 2024-08-19T14:19:11.231 INFO:tasks.workunit.client.0.smithi102.stderr:54007: instructing pg 1.1b on osd.0 to scrub 2024-08-19T14:19:11.231 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:11.231 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1b") | .state' 2024-08-19T14:19:11.231 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:11.231 INFO:tasks.workunit.client.0.smithi102.stderr:54007: dumped pgs 2024-08-19T14:19:11.232 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1b 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.232 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1b 2024-08-19T14:19:11.232 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.232 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:11.232 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:11.232 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:11.232 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:11.232 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:11.232 INFO:tasks.workunit.client.0.smithi102.stderr:54007: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1b last_scrub_stamp 2024-08-19T14:19:11.232 INFO:tasks.workunit.client.0.smithi102.stderr:54007: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-08-19T14:19:11.232 INFO:tasks.workunit.client.0.smithi102.stderr:54007: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.232 INFO:tasks.workunit.client.0.smithi102.stderr:54007: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.232 INFO:tasks.workunit.client.0.smithi102.stderr:54007: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-08-19T14:19:11.233 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:11.233 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:11.233 INFO:tasks.workunit.client.0.smithi102.stderr:54007: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:11.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:11.233 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1e 2024-08-19T14:19:11.233 INFO:tasks.workunit.client.0.smithi102.stderr:54079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1e 2024-08-19T14:19:11.233 INFO:tasks.workunit.client.0.smithi102.stderr:54079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-08-19T14:19:11.233 INFO:tasks.workunit.client.0.smithi102.stderr:54079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.234 INFO:tasks.workunit.client.0.smithi102.stderr:54079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.234 INFO:tasks.workunit.client.0.smithi102.stderr:54079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-08-19T14:19:11.234 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.234 INFO:tasks.workunit.client.0.smithi102.stderr:54079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1e query 2024-08-19T14:19:11.234 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:11.234 INFO:tasks.workunit.client.0.smithi102.stderr:54079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:11.234 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:11.234 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:11.234 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1e 2024-08-19T14:19:11.234 INFO:tasks.workunit.client.0.smithi102.stderr:54079: instructing pg 1.1e on osd.3 to scrub 2024-08-19T14:19:11.234 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:11.234 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1e") | .state' 2024-08-19T14:19:11.234 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:11.235 INFO:tasks.workunit.client.0.smithi102.stderr:54079: dumped pgs 2024-08-19T14:19:11.235 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1e 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.235 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1e 2024-08-19T14:19:11.235 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.235 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:11.235 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:11.235 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:11.235 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:11.235 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:11.235 INFO:tasks.workunit.client.0.smithi102.stderr:54079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1e last_scrub_stamp 2024-08-19T14:19:11.235 INFO:tasks.workunit.client.0.smithi102.stderr:54079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-08-19T14:19:11.235 INFO:tasks.workunit.client.0.smithi102.stderr:54079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.235 INFO:tasks.workunit.client.0.smithi102.stderr:54079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.236 INFO:tasks.workunit.client.0.smithi102.stderr:54079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-08-19T14:19:11.236 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:11.236 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:11.236 INFO:tasks.workunit.client.0.smithi102.stderr:54079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:11.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:11.247 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.10 2024-08-19T14:19:11.247 INFO:tasks.workunit.client.0.smithi102.stderr:53859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.10 2024-08-19T14:19:11.247 INFO:tasks.workunit.client.0.smithi102.stderr:53859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-08-19T14:19:11.247 INFO:tasks.workunit.client.0.smithi102.stderr:53859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.247 INFO:tasks.workunit.client.0.smithi102.stderr:53859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.247 INFO:tasks.workunit.client.0.smithi102.stderr:53859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-08-19T14:19:11.247 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.247 INFO:tasks.workunit.client.0.smithi102.stderr:53859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.10 query 2024-08-19T14:19:11.247 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:11.247 INFO:tasks.workunit.client.0.smithi102.stderr:53859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:11.247 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:11.248 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:11.248 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.10 2024-08-19T14:19:11.248 INFO:tasks.workunit.client.0.smithi102.stderr:53859: instructing pg 1.10 on osd.5 to scrub 2024-08-19T14:19:11.248 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.10") | .state' 2024-08-19T14:19:11.248 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:11.248 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:11.248 INFO:tasks.workunit.client.0.smithi102.stderr:53859: dumped pgs 2024-08-19T14:19:11.248 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.10 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.248 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.10 2024-08-19T14:19:11.248 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.248 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:11.248 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:11.248 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:11.249 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:11.249 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:11.249 INFO:tasks.workunit.client.0.smithi102.stderr:53859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.10 last_scrub_stamp 2024-08-19T14:19:11.249 INFO:tasks.workunit.client.0.smithi102.stderr:53859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-08-19T14:19:11.249 INFO:tasks.workunit.client.0.smithi102.stderr:53859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.249 INFO:tasks.workunit.client.0.smithi102.stderr:53859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-08-19T14:19:11.249 INFO:tasks.workunit.client.0.smithi102.stderr:53859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.249 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:11.249 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:11.249 INFO:tasks.workunit.client.0.smithi102.stderr:53859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:11.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:11.270 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.0 2024-08-19T14:19:11.270 INFO:tasks.workunit.client.0.smithi102.stderr:53747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.0 2024-08-19T14:19:11.270 INFO:tasks.workunit.client.0.smithi102.stderr:53747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:19:11.270 INFO:tasks.workunit.client.0.smithi102.stderr:53747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.270 INFO:tasks.workunit.client.0.smithi102.stderr:53747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.270 INFO:tasks.workunit.client.0.smithi102.stderr:53747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:19:11.270 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.270 INFO:tasks.workunit.client.0.smithi102.stderr:53747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:11.271 INFO:tasks.workunit.client.0.smithi102.stderr:53747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.0 query 2024-08-19T14:19:11.271 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:11.271 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:11.271 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:11.271 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.0 2024-08-19T14:19:11.271 INFO:tasks.workunit.client.0.smithi102.stderr:53747: instructing pg 1.0 on osd.7 to scrub 2024-08-19T14:19:11.271 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.0") | .state' 2024-08-19T14:19:11.271 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:11.271 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:11.271 INFO:tasks.workunit.client.0.smithi102.stderr:53747: dumped pgs 2024-08-19T14:19:11.271 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.0 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.271 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.0 2024-08-19T14:19:11.272 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.272 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:11.272 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:11.272 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:11.272 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:11.272 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:11.272 INFO:tasks.workunit.client.0.smithi102.stderr:53747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:19:11.272 INFO:tasks.workunit.client.0.smithi102.stderr:53747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:19:11.272 INFO:tasks.workunit.client.0.smithi102.stderr:53747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.272 INFO:tasks.workunit.client.0.smithi102.stderr:53747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.272 INFO:tasks.workunit.client.0.smithi102.stderr:53747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:19:11.272 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:11.273 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:11.273 INFO:tasks.workunit.client.0.smithi102.stderr:53747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:11.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:11.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:11.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:11.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:11.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:11.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53749 2024-08-19T14:19:11.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:11.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:11.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:11.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:11.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53756 2024-08-19T14:19:11.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:11.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:11.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:11.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:11.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53764 2024-08-19T14:19:11.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:11.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:11.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:11.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:11.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53769 2024-08-19T14:19:11.291 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.f 2024-08-19T14:19:11.291 INFO:tasks.workunit.client.0.smithi102.stderr:53849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.f 2024-08-19T14:19:11.291 INFO:tasks.workunit.client.0.smithi102.stderr:53849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-08-19T14:19:11.291 INFO:tasks.workunit.client.0.smithi102.stderr:53849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.291 INFO:tasks.workunit.client.0.smithi102.stderr:53849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-08-19T14:19:11.292 INFO:tasks.workunit.client.0.smithi102.stderr:53849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.292 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.292 INFO:tasks.workunit.client.0.smithi102.stderr:53849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.f query 2024-08-19T14:19:11.292 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:11.292 INFO:tasks.workunit.client.0.smithi102.stderr:53849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:11.292 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:11.292 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:11.292 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.f 2024-08-19T14:19:11.292 INFO:tasks.workunit.client.0.smithi102.stderr:53849: instructing pg 1.f on osd.6 to scrub 2024-08-19T14:19:11.292 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:11.292 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.f") | .state' 2024-08-19T14:19:11.292 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:11.292 INFO:tasks.workunit.client.0.smithi102.stderr:53849: dumped pgs 2024-08-19T14:19:11.293 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.f 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.293 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.f 2024-08-19T14:19:11.293 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.293 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:11.293 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:11.293 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:11.294 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:11.294 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:11.294 INFO:tasks.workunit.client.0.smithi102.stderr:53849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.f last_scrub_stamp 2024-08-19T14:19:11.294 INFO:tasks.workunit.client.0.smithi102.stderr:53849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-08-19T14:19:11.294 INFO:tasks.workunit.client.0.smithi102.stderr:53849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.294 INFO:tasks.workunit.client.0.smithi102.stderr:53849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-08-19T14:19:11.294 INFO:tasks.workunit.client.0.smithi102.stderr:53849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.295 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:11.295 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:11.295 INFO:tasks.workunit.client.0.smithi102.stderr:53849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:11.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:11.352 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.b 2024-08-19T14:19:11.352 INFO:tasks.workunit.client.0.smithi102.stderr:53820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.b 2024-08-19T14:19:11.352 INFO:tasks.workunit.client.0.smithi102.stderr:53820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-08-19T14:19:11.352 INFO:tasks.workunit.client.0.smithi102.stderr:53820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.352 INFO:tasks.workunit.client.0.smithi102.stderr:53820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.353 INFO:tasks.workunit.client.0.smithi102.stderr:53820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-08-19T14:19:11.353 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.353 INFO:tasks.workunit.client.0.smithi102.stderr:53820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.b query 2024-08-19T14:19:11.353 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:11.353 INFO:tasks.workunit.client.0.smithi102.stderr:53820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:11.353 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:11.353 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:11.353 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.b 2024-08-19T14:19:11.353 INFO:tasks.workunit.client.0.smithi102.stderr:53820: instructing pg 1.b on osd.1 to scrub 2024-08-19T14:19:11.353 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:11.353 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:11.353 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.b") | .state' 2024-08-19T14:19:11.354 INFO:tasks.workunit.client.0.smithi102.stderr:53820: dumped pgs 2024-08-19T14:19:11.354 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.b 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.354 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.b 2024-08-19T14:19:11.354 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.354 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:11.354 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:11.354 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:11.354 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:11.354 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:11.354 INFO:tasks.workunit.client.0.smithi102.stderr:53820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.b last_scrub_stamp 2024-08-19T14:19:11.354 INFO:tasks.workunit.client.0.smithi102.stderr:53820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-08-19T14:19:11.354 INFO:tasks.workunit.client.0.smithi102.stderr:53820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.354 INFO:tasks.workunit.client.0.smithi102.stderr:53820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.355 INFO:tasks.workunit.client.0.smithi102.stderr:53820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-08-19T14:19:11.355 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:11.355 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:11.355 INFO:tasks.workunit.client.0.smithi102.stderr:53820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:11.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:11.422 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.9 2024-08-19T14:19:11.422 INFO:tasks.workunit.client.0.smithi102.stderr:53810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.9 2024-08-19T14:19:11.422 INFO:tasks.workunit.client.0.smithi102.stderr:53810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-08-19T14:19:11.422 INFO:tasks.workunit.client.0.smithi102.stderr:53810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.422 INFO:tasks.workunit.client.0.smithi102.stderr:53810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.422 INFO:tasks.workunit.client.0.smithi102.stderr:53810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-08-19T14:19:11.422 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.422 INFO:tasks.workunit.client.0.smithi102.stderr:53810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.9 query 2024-08-19T14:19:11.423 INFO:tasks.workunit.client.0.smithi102.stderr:53810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:11.423 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:11.423 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:11.423 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:11.423 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.9 2024-08-19T14:19:11.423 INFO:tasks.workunit.client.0.smithi102.stderr:53810: instructing pg 1.9 on osd.1 to scrub 2024-08-19T14:19:11.423 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:11.423 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.9") | .state' 2024-08-19T14:19:11.423 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:11.423 INFO:tasks.workunit.client.0.smithi102.stderr:53810: dumped pgs 2024-08-19T14:19:11.423 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.9 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.423 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.9 2024-08-19T14:19:11.423 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.424 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:11.424 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:11.424 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:11.424 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:11.424 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:11.424 INFO:tasks.workunit.client.0.smithi102.stderr:53810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.9 last_scrub_stamp 2024-08-19T14:19:11.424 INFO:tasks.workunit.client.0.smithi102.stderr:53810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-08-19T14:19:11.424 INFO:tasks.workunit.client.0.smithi102.stderr:53810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.424 INFO:tasks.workunit.client.0.smithi102.stderr:53810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.424 INFO:tasks.workunit.client.0.smithi102.stderr:53810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-08-19T14:19:11.425 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:11.425 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:11.425 INFO:tasks.workunit.client.0.smithi102.stderr:53810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:11.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:11.489 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1a 2024-08-19T14:19:11.489 INFO:tasks.workunit.client.0.smithi102.stderr:54001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1a 2024-08-19T14:19:11.489 INFO:tasks.workunit.client.0.smithi102.stderr:54001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-08-19T14:19:11.489 INFO:tasks.workunit.client.0.smithi102.stderr:54001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.489 INFO:tasks.workunit.client.0.smithi102.stderr:54001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.489 INFO:tasks.workunit.client.0.smithi102.stderr:54001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-08-19T14:19:11.489 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.489 INFO:tasks.workunit.client.0.smithi102.stderr:54001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1a query 2024-08-19T14:19:11.489 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:11.489 INFO:tasks.workunit.client.0.smithi102.stderr:54001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:11.490 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:11.490 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:11.490 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1a 2024-08-19T14:19:11.490 INFO:tasks.workunit.client.0.smithi102.stderr:54001: instructing pg 1.1a on osd.1 to scrub 2024-08-19T14:19:11.490 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:11.490 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:11.490 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1a") | .state' 2024-08-19T14:19:11.490 INFO:tasks.workunit.client.0.smithi102.stderr:54001: dumped pgs 2024-08-19T14:19:11.490 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1a 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.490 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1a 2024-08-19T14:19:11.490 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.490 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:11.490 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:11.491 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:11.491 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:11.491 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:11.491 INFO:tasks.workunit.client.0.smithi102.stderr:54001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1a last_scrub_stamp 2024-08-19T14:19:11.491 INFO:tasks.workunit.client.0.smithi102.stderr:54001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-08-19T14:19:11.491 INFO:tasks.workunit.client.0.smithi102.stderr:54001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.491 INFO:tasks.workunit.client.0.smithi102.stderr:54001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.491 INFO:tasks.workunit.client.0.smithi102.stderr:54001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-08-19T14:19:11.491 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:19:07.426089+0000 '>' '' 2024-08-19T14:19:11.491 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:11.491 INFO:tasks.workunit.client.0.smithi102.stderr:54001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:11.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:11.599 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.7 2024-08-19T14:19:11.599 INFO:tasks.workunit.client.0.smithi102.stderr:53793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.7 2024-08-19T14:19:11.599 INFO:tasks.workunit.client.0.smithi102.stderr:53793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-08-19T14:19:11.599 INFO:tasks.workunit.client.0.smithi102.stderr:53793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.599 INFO:tasks.workunit.client.0.smithi102.stderr:53793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-08-19T14:19:11.599 INFO:tasks.workunit.client.0.smithi102.stderr:53793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.599 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.599 INFO:tasks.workunit.client.0.smithi102.stderr:53793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.7 query 2024-08-19T14:19:11.599 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:11.599 INFO:tasks.workunit.client.0.smithi102.stderr:53793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:11.600 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:11.600 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:11.600 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.7 2024-08-19T14:19:11.600 INFO:tasks.workunit.client.0.smithi102.stderr:53793: instructing pg 1.7 on osd.6 to scrub 2024-08-19T14:19:11.600 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:11.600 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.7") | .state' 2024-08-19T14:19:11.600 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:11.600 INFO:tasks.workunit.client.0.smithi102.stderr:53793: dumped pgs 2024-08-19T14:19:11.600 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.7 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.600 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.7 2024-08-19T14:19:11.600 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.600 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:11.600 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:11.601 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:11.601 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:11.601 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:11.601 INFO:tasks.workunit.client.0.smithi102.stderr:53793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-08-19T14:19:11.601 INFO:tasks.workunit.client.0.smithi102.stderr:53793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-08-19T14:19:11.601 INFO:tasks.workunit.client.0.smithi102.stderr:53793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.601 INFO:tasks.workunit.client.0.smithi102.stderr:53793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-08-19T14:19:11.601 INFO:tasks.workunit.client.0.smithi102.stderr:53793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.601 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:11.601 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:11.601 INFO:tasks.workunit.client.0.smithi102.stderr:53793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:11.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:11.630 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.16 2024-08-19T14:19:11.630 INFO:tasks.workunit.client.0.smithi102.stderr:53925: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.16 2024-08-19T14:19:11.630 INFO:tasks.workunit.client.0.smithi102.stderr:53925: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-08-19T14:19:11.630 INFO:tasks.workunit.client.0.smithi102.stderr:53925: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.630 INFO:tasks.workunit.client.0.smithi102.stderr:53925: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.631 INFO:tasks.workunit.client.0.smithi102.stderr:53925: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-08-19T14:19:11.631 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.631 INFO:tasks.workunit.client.0.smithi102.stderr:53925: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.16 query 2024-08-19T14:19:11.631 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:11.631 INFO:tasks.workunit.client.0.smithi102.stderr:53925: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:11.631 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:11.631 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:11.631 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.16 2024-08-19T14:19:11.631 INFO:tasks.workunit.client.0.smithi102.stderr:53925: instructing pg 1.16 on osd.6 to scrub 2024-08-19T14:19:11.631 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:11.631 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:11.631 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.16") | .state' 2024-08-19T14:19:11.631 INFO:tasks.workunit.client.0.smithi102.stderr:53925: dumped pgs 2024-08-19T14:19:11.632 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.16 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.632 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.16 2024-08-19T14:19:11.632 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.632 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:11.632 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:11.632 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:11.632 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:11.632 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:11.632 INFO:tasks.workunit.client.0.smithi102.stderr:53925: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.16 last_scrub_stamp 2024-08-19T14:19:11.632 INFO:tasks.workunit.client.0.smithi102.stderr:53925: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-08-19T14:19:11.632 INFO:tasks.workunit.client.0.smithi102.stderr:53925: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.632 INFO:tasks.workunit.client.0.smithi102.stderr:53925: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-08-19T14:19:11.632 INFO:tasks.workunit.client.0.smithi102.stderr:53925: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.633 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:19:08.910844+0000 '>' '' 2024-08-19T14:19:11.633 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:11.633 INFO:tasks.workunit.client.0.smithi102.stderr:53925: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:11.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:11.743 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.13 2024-08-19T14:19:11.743 INFO:tasks.workunit.client.0.smithi102.stderr:53893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.13 2024-08-19T14:19:11.743 INFO:tasks.workunit.client.0.smithi102.stderr:53893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-08-19T14:19:11.743 INFO:tasks.workunit.client.0.smithi102.stderr:53893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.743 INFO:tasks.workunit.client.0.smithi102.stderr:53893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-08-19T14:19:11.743 INFO:tasks.workunit.client.0.smithi102.stderr:53893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.743 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.743 INFO:tasks.workunit.client.0.smithi102.stderr:53893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:11.743 INFO:tasks.workunit.client.0.smithi102.stderr:53893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.13 query 2024-08-19T14:19:11.743 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:11.743 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:11.744 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:11.744 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.13 2024-08-19T14:19:11.744 INFO:tasks.workunit.client.0.smithi102.stderr:53893: instructing pg 1.13 on osd.4 to scrub 2024-08-19T14:19:11.744 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:11.744 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:11.744 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.13") | .state' 2024-08-19T14:19:11.744 INFO:tasks.workunit.client.0.smithi102.stderr:53893: dumped pgs 2024-08-19T14:19:11.744 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.13 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.744 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.13 2024-08-19T14:19:11.744 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.744 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:11.744 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:11.744 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:11.745 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:11.745 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:11.745 INFO:tasks.workunit.client.0.smithi102.stderr:53893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.13 last_scrub_stamp 2024-08-19T14:19:11.745 INFO:tasks.workunit.client.0.smithi102.stderr:53893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-08-19T14:19:11.745 INFO:tasks.workunit.client.0.smithi102.stderr:53893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.745 INFO:tasks.workunit.client.0.smithi102.stderr:53893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-08-19T14:19:11.745 INFO:tasks.workunit.client.0.smithi102.stderr:53893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.745 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:11.745 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:11.745 INFO:tasks.workunit.client.0.smithi102.stderr:53893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:11.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:11.768 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.19 2024-08-19T14:19:11.768 INFO:tasks.workunit.client.0.smithi102.stderr:53961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.19 2024-08-19T14:19:11.768 INFO:tasks.workunit.client.0.smithi102.stderr:53961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-08-19T14:19:11.768 INFO:tasks.workunit.client.0.smithi102.stderr:53961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.768 INFO:tasks.workunit.client.0.smithi102.stderr:53961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-08-19T14:19:11.769 INFO:tasks.workunit.client.0.smithi102.stderr:53961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.769 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.769 INFO:tasks.workunit.client.0.smithi102.stderr:53961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:11.769 INFO:tasks.workunit.client.0.smithi102.stderr:53961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.19 query 2024-08-19T14:19:11.769 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:11.769 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:11.769 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:11.769 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.19 2024-08-19T14:19:11.769 INFO:tasks.workunit.client.0.smithi102.stderr:53961: instructing pg 1.19 on osd.3 to scrub 2024-08-19T14:19:11.769 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:11.769 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.19") | .state' 2024-08-19T14:19:11.769 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:11.769 INFO:tasks.workunit.client.0.smithi102.stderr:53961: dumped pgs 2024-08-19T14:19:11.770 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.19 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.770 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.19 2024-08-19T14:19:11.770 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.770 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:11.770 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:11.770 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:11.770 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:11.770 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:11.770 INFO:tasks.workunit.client.0.smithi102.stderr:53961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.19 last_scrub_stamp 2024-08-19T14:19:11.771 INFO:tasks.workunit.client.0.smithi102.stderr:53961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-08-19T14:19:11.771 INFO:tasks.workunit.client.0.smithi102.stderr:53961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.771 INFO:tasks.workunit.client.0.smithi102.stderr:53961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.771 INFO:tasks.workunit.client.0.smithi102.stderr:53961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-08-19T14:19:11.771 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:11.771 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:11.771 INFO:tasks.workunit.client.0.smithi102.stderr:53961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:11.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:11.812 INFO:tasks.workunit.client.0.smithi102.stderr:_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.812 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:11.812 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:11.812 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-08-19T14:19:11.812 INFO:tasks.workunit.client.0.smithi102.stderr:53785: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-08-19T14:19:11.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-08-19T14:19:11.829 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1d 2024-08-19T14:19:11.829 INFO:tasks.workunit.client.0.smithi102.stderr:54074: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1d 2024-08-19T14:19:11.829 INFO:tasks.workunit.client.0.smithi102.stderr:54074: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-08-19T14:19:11.830 INFO:tasks.workunit.client.0.smithi102.stderr:54074: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.830 INFO:tasks.workunit.client.0.smithi102.stderr:54074: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.830 INFO:tasks.workunit.client.0.smithi102.stderr:54074: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-08-19T14:19:11.830 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.830 INFO:tasks.workunit.client.0.smithi102.stderr:54074: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:11.830 INFO:tasks.workunit.client.0.smithi102.stderr:54074: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1d query 2024-08-19T14:19:11.830 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:11.830 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:11.830 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:11.830 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1d 2024-08-19T14:19:11.831 INFO:tasks.workunit.client.0.smithi102.stderr:54074: instructing pg 1.1d on osd.1 to scrub 2024-08-19T14:19:11.831 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:11.831 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:11.831 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1d") | .state' 2024-08-19T14:19:11.831 INFO:tasks.workunit.client.0.smithi102.stderr:54074: dumped pgs 2024-08-19T14:19:11.831 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1d 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.831 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1d 2024-08-19T14:19:11.831 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.831 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:11.831 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:11.831 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:11.831 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:11.831 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:11.832 INFO:tasks.workunit.client.0.smithi102.stderr:54074: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1d last_scrub_stamp 2024-08-19T14:19:11.832 INFO:tasks.workunit.client.0.smithi102.stderr:54074: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-08-19T14:19:11.832 INFO:tasks.workunit.client.0.smithi102.stderr:54074: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.832 INFO:tasks.workunit.client.0.smithi102.stderr:54074: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-08-19T14:19:11.832 INFO:tasks.workunit.client.0.smithi102.stderr:54074: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.832 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:11.832 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:11.832 INFO:tasks.workunit.client.0.smithi102.stderr:54074: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:11.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:11.871 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.8 2024-08-19T14:19:11.871 INFO:tasks.workunit.client.0.smithi102.stderr:53805: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.8 2024-08-19T14:19:11.871 INFO:tasks.workunit.client.0.smithi102.stderr:53805: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-08-19T14:19:11.871 INFO:tasks.workunit.client.0.smithi102.stderr:53805: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.871 INFO:tasks.workunit.client.0.smithi102.stderr:53805: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.871 INFO:tasks.workunit.client.0.smithi102.stderr:53805: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-08-19T14:19:11.871 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.871 INFO:tasks.workunit.client.0.smithi102.stderr:53805: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:11.872 INFO:tasks.workunit.client.0.smithi102.stderr:53805: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.8 query 2024-08-19T14:19:11.872 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:11.872 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:11.872 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:11.872 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.8 2024-08-19T14:19:11.872 INFO:tasks.workunit.client.0.smithi102.stderr:53805: instructing pg 1.8 on osd.5 to scrub 2024-08-19T14:19:11.872 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:11.872 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:11.872 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.8") | .state' 2024-08-19T14:19:11.872 INFO:tasks.workunit.client.0.smithi102.stderr:53805: dumped pgs 2024-08-19T14:19:11.872 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.8 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.872 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.8 2024-08-19T14:19:11.872 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:11.873 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:11.873 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:11.873 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:11.873 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:11.873 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:11.873 INFO:tasks.workunit.client.0.smithi102.stderr:53805: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.8 last_scrub_stamp 2024-08-19T14:19:11.873 INFO:tasks.workunit.client.0.smithi102.stderr:53805: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-08-19T14:19:11.873 INFO:tasks.workunit.client.0.smithi102.stderr:53805: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:11.873 INFO:tasks.workunit.client.0.smithi102.stderr:53805: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:11.873 INFO:tasks.workunit.client.0.smithi102.stderr:53805: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-08-19T14:19:11.874 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:11.874 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:11.874 INFO:tasks.workunit.client.0.smithi102.stderr:53805: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:11.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:12.022 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.11 2024-08-19T14:19:12.022 INFO:tasks.workunit.client.0.smithi102.stderr:53878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.11 2024-08-19T14:19:12.022 INFO:tasks.workunit.client.0.smithi102.stderr:53878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-08-19T14:19:12.022 INFO:tasks.workunit.client.0.smithi102.stderr:53878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.022 INFO:tasks.workunit.client.0.smithi102.stderr:53878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-08-19T14:19:12.023 INFO:tasks.workunit.client.0.smithi102.stderr:53878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.023 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.023 INFO:tasks.workunit.client.0.smithi102.stderr:53878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.11 query 2024-08-19T14:19:12.023 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:12.023 INFO:tasks.workunit.client.0.smithi102.stderr:53878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:12.023 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:12.023 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:12.023 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.11 2024-08-19T14:19:12.023 INFO:tasks.workunit.client.0.smithi102.stderr:53878: instructing pg 1.11 on osd.0 to scrub 2024-08-19T14:19:12.023 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.11") | .state' 2024-08-19T14:19:12.023 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:12.023 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:12.023 INFO:tasks.workunit.client.0.smithi102.stderr:53878: dumped pgs 2024-08-19T14:19:12.023 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.11 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.024 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.11 2024-08-19T14:19:12.024 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.024 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:12.024 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:12.024 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:12.024 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:12.024 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:12.024 INFO:tasks.workunit.client.0.smithi102.stderr:53878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.11 last_scrub_stamp 2024-08-19T14:19:12.024 INFO:tasks.workunit.client.0.smithi102.stderr:53878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-08-19T14:19:12.024 INFO:tasks.workunit.client.0.smithi102.stderr:53878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.024 INFO:tasks.workunit.client.0.smithi102.stderr:53878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-08-19T14:19:12.024 INFO:tasks.workunit.client.0.smithi102.stderr:53878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.024 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:12.024 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:12.025 INFO:tasks.workunit.client.0.smithi102.stderr:53878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:12.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:12.080 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.14 2024-08-19T14:19:12.080 INFO:tasks.workunit.client.0.smithi102.stderr:53901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.14 2024-08-19T14:19:12.081 INFO:tasks.workunit.client.0.smithi102.stderr:53901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-08-19T14:19:12.081 INFO:tasks.workunit.client.0.smithi102.stderr:53901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.081 INFO:tasks.workunit.client.0.smithi102.stderr:53901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-08-19T14:19:12.081 INFO:tasks.workunit.client.0.smithi102.stderr:53901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.081 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.081 INFO:tasks.workunit.client.0.smithi102.stderr:53901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.14 query 2024-08-19T14:19:12.081 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:12.081 INFO:tasks.workunit.client.0.smithi102.stderr:53901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:12.081 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:12.081 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:12.081 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.14 2024-08-19T14:19:12.081 INFO:tasks.workunit.client.0.smithi102.stderr:53901: instructing pg 1.14 on osd.2 to scrub 2024-08-19T14:19:12.081 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:12.082 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:12.082 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.14") | .state' 2024-08-19T14:19:12.082 INFO:tasks.workunit.client.0.smithi102.stderr:53901: dumped pgs 2024-08-19T14:19:12.082 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.14 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.082 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.14 2024-08-19T14:19:12.082 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.082 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:12.082 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:12.082 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:12.082 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:12.082 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:12.082 INFO:tasks.workunit.client.0.smithi102.stderr:53901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.14 last_scrub_stamp 2024-08-19T14:19:12.082 INFO:tasks.workunit.client.0.smithi102.stderr:53901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-08-19T14:19:12.083 INFO:tasks.workunit.client.0.smithi102.stderr:53901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.083 INFO:tasks.workunit.client.0.smithi102.stderr:53901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.083 INFO:tasks.workunit.client.0.smithi102.stderr:53901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-08-19T14:19:12.083 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:19:09.104181+0000 '>' '' 2024-08-19T14:19:12.083 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:12.083 INFO:tasks.workunit.client.0.smithi102.stderr:53901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:12.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:12.097 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1c 2024-08-19T14:19:12.098 INFO:tasks.workunit.client.0.smithi102.stderr:54024: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1c 2024-08-19T14:19:12.098 INFO:tasks.workunit.client.0.smithi102.stderr:54024: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-08-19T14:19:12.098 INFO:tasks.workunit.client.0.smithi102.stderr:54024: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.098 INFO:tasks.workunit.client.0.smithi102.stderr:54024: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-08-19T14:19:12.098 INFO:tasks.workunit.client.0.smithi102.stderr:54024: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.098 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.098 INFO:tasks.workunit.client.0.smithi102.stderr:54024: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1c query 2024-08-19T14:19:12.098 INFO:tasks.workunit.client.0.smithi102.stderr:54024: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:12.098 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:12.098 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:12.099 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:12.099 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1c 2024-08-19T14:19:12.099 INFO:tasks.workunit.client.0.smithi102.stderr:54024: instructing pg 1.1c on osd.0 to scrub 2024-08-19T14:19:12.099 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:12.099 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1c") | .state' 2024-08-19T14:19:12.099 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:12.099 INFO:tasks.workunit.client.0.smithi102.stderr:54024: dumped pgs 2024-08-19T14:19:12.099 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1c 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.099 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1c 2024-08-19T14:19:12.099 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.099 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:12.099 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:12.099 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:12.100 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:12.100 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:12.100 INFO:tasks.workunit.client.0.smithi102.stderr:54024: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1c last_scrub_stamp 2024-08-19T14:19:12.100 INFO:tasks.workunit.client.0.smithi102.stderr:54024: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-08-19T14:19:12.100 INFO:tasks.workunit.client.0.smithi102.stderr:54024: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.100 INFO:tasks.workunit.client.0.smithi102.stderr:54024: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-08-19T14:19:12.100 INFO:tasks.workunit.client.0.smithi102.stderr:54024: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.100 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:12.100 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:12.100 INFO:tasks.workunit.client.0.smithi102.stderr:54024: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:12.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:12.114 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.15 2024-08-19T14:19:12.114 INFO:tasks.workunit.client.0.smithi102.stderr:53906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.15 2024-08-19T14:19:12.114 INFO:tasks.workunit.client.0.smithi102.stderr:53906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-08-19T14:19:12.114 INFO:tasks.workunit.client.0.smithi102.stderr:53906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.114 INFO:tasks.workunit.client.0.smithi102.stderr:53906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-08-19T14:19:12.115 INFO:tasks.workunit.client.0.smithi102.stderr:53906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.115 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.115 INFO:tasks.workunit.client.0.smithi102.stderr:53906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.15 query 2024-08-19T14:19:12.115 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:12.115 INFO:tasks.workunit.client.0.smithi102.stderr:53906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:12.115 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:12.115 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:12.115 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.15 2024-08-19T14:19:12.115 INFO:tasks.workunit.client.0.smithi102.stderr:53906: instructing pg 1.15 on osd.1 to scrub 2024-08-19T14:19:12.115 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.15") | .state' 2024-08-19T14:19:12.115 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:12.115 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:12.115 INFO:tasks.workunit.client.0.smithi102.stderr:53906: dumped pgs 2024-08-19T14:19:12.116 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.15 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.116 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.15 2024-08-19T14:19:12.116 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.116 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:12.116 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:12.116 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:12.116 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:12.116 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:12.116 INFO:tasks.workunit.client.0.smithi102.stderr:53906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.15 last_scrub_stamp 2024-08-19T14:19:12.116 INFO:tasks.workunit.client.0.smithi102.stderr:53906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-08-19T14:19:12.116 INFO:tasks.workunit.client.0.smithi102.stderr:53906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.116 INFO:tasks.workunit.client.0.smithi102.stderr:53906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-08-19T14:19:12.116 INFO:tasks.workunit.client.0.smithi102.stderr:53906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.117 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:12.117 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:12.117 INFO:tasks.workunit.client.0.smithi102.stderr:53906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:12.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:12.143 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.4 2024-08-19T14:19:12.144 INFO:tasks.workunit.client.0.smithi102.stderr:53774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.4 2024-08-19T14:19:12.144 INFO:tasks.workunit.client.0.smithi102.stderr:53774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-08-19T14:19:12.144 INFO:tasks.workunit.client.0.smithi102.stderr:53774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.144 INFO:tasks.workunit.client.0.smithi102.stderr:53774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-08-19T14:19:12.144 INFO:tasks.workunit.client.0.smithi102.stderr:53774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.144 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.144 INFO:tasks.workunit.client.0.smithi102.stderr:53774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.4 query 2024-08-19T14:19:12.144 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:12.144 INFO:tasks.workunit.client.0.smithi102.stderr:53774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:12.144 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:12.144 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:12.144 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.4 2024-08-19T14:19:12.145 INFO:tasks.workunit.client.0.smithi102.stderr:53774: instructing pg 1.4 on osd.3 to scrub 2024-08-19T14:19:12.145 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:12.145 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:12.145 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.4") | .state' 2024-08-19T14:19:12.145 INFO:tasks.workunit.client.0.smithi102.stderr:53774: dumped pgs 2024-08-19T14:19:12.145 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.4 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.145 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.4 2024-08-19T14:19:12.145 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.145 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:12.145 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:12.145 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:12.145 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:12.145 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:12.146 INFO:tasks.workunit.client.0.smithi102.stderr:53774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-08-19T14:19:12.146 INFO:tasks.workunit.client.0.smithi102.stderr:53774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-08-19T14:19:12.146 INFO:tasks.workunit.client.0.smithi102.stderr:53774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.146 INFO:tasks.workunit.client.0.smithi102.stderr:53774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.146 INFO:tasks.workunit.client.0.smithi102.stderr:53774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-08-19T14:19:12.146 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:12.146 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:12.146 INFO:tasks.workunit.client.0.smithi102.stderr:53774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:12.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:12.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53773 2024-08-19T14:19:12.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53782 2024-08-19T14:19:12.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-08-19T14:19:12.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-08-19T14:19:12.151 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 0 + 1 2024-08-19T14:19:12.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=1 2024-08-19T14:19:12.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53786 2024-08-19T14:19:12.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53800 2024-08-19T14:19:12.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53802 2024-08-19T14:19:12.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53806 2024-08-19T14:19:12.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-08-19T14:19:12.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-08-19T14:19:12.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 1 + 1 2024-08-19T14:19:12.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=2 2024-08-19T14:19:12.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53814 2024-08-19T14:19:12.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53821 2024-08-19T14:19:12.189 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.c 2024-08-19T14:19:12.189 INFO:tasks.workunit.client.0.smithi102.stderr:53831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.c 2024-08-19T14:19:12.189 INFO:tasks.workunit.client.0.smithi102.stderr:53831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-08-19T14:19:12.189 INFO:tasks.workunit.client.0.smithi102.stderr:53831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.190 INFO:tasks.workunit.client.0.smithi102.stderr:53831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-08-19T14:19:12.190 INFO:tasks.workunit.client.0.smithi102.stderr:53831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.190 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.190 INFO:tasks.workunit.client.0.smithi102.stderr:53831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:12.190 INFO:tasks.workunit.client.0.smithi102.stderr:53831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.c query 2024-08-19T14:19:12.190 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:12.190 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:12.190 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:12.190 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.c 2024-08-19T14:19:12.191 INFO:tasks.workunit.client.0.smithi102.stderr:53831: instructing pg 1.c on osd.3 to scrub 2024-08-19T14:19:12.191 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:12.191 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.c") | .state' 2024-08-19T14:19:12.191 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:12.191 INFO:tasks.workunit.client.0.smithi102.stderr:53831: dumped pgs 2024-08-19T14:19:12.191 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.c 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.191 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.c 2024-08-19T14:19:12.191 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.191 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:12.191 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:12.192 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:12.192 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:12.192 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:12.192 INFO:tasks.workunit.client.0.smithi102.stderr:53831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.c last_scrub_stamp 2024-08-19T14:19:12.192 INFO:tasks.workunit.client.0.smithi102.stderr:53831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-08-19T14:19:12.192 INFO:tasks.workunit.client.0.smithi102.stderr:53831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.192 INFO:tasks.workunit.client.0.smithi102.stderr:53831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.192 INFO:tasks.workunit.client.0.smithi102.stderr:53831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-08-19T14:19:12.192 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scr53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.d 2024-08-19T14:19:12.192 INFO:tasks.workunit.client.0.smithi102.stderr:53838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.d 2024-08-19T14:19:12.192 INFO:tasks.workunit.client.0.smithi102.stderr:53838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-08-19T14:19:12.192 INFO:tasks.workunit.client.0.smithi102.stderr:53838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.192 INFO:tasks.workunit.client.0.smithi102.stderr:53838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.193 INFO:tasks.workunit.client.0.smithi102.stderr:53838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-08-19T14:19:12.193 INFO:tasks.workunit.client.0.smithi102.stderr:53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.193 INFO:tasks.workunit.client.0.smithi102.stderr:53838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:12.193 INFO:tasks.workunit.client.0.smithi102.stderr:53838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.d query 2024-08-19T14:19:12.193 INFO:tasks.workunit.client.0.smithi102.stderr:53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:12.193 INFO:tasks.workunit.client.0.smithi102.stderr:53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:12.193 INFO:tasks.workunit.client.0.smithi102.stderr:53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:12.193 INFO:tasks.workunit.client.0.smithi102.stderr:53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.d 2024-08-19T14:19:12.193 INFO:tasks.workunit.client.0.smithi102.stderr:53838: instructing pg 1.d on osd.7 to scrub 2024-08-19T14:19:12.193 INFO:tasks.workunit.client.0.smithi102.stderr:53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:12.193 INFO:tasks.workunit.client.0.smithi102.stderr:53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.d") | .state' 2024-08-19T14:19:12.193 INFO:tasks.workunit.client.0.smithi102.stderr:53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:12.193 INFO:tasks.workunit.client.0.smithi102.stderr:53838: dumped pgs 2024-08-19T14:19:12.194 INFO:tasks.workunit.client.0.smithi102.stderr:53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.d 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.194 INFO:tasks.workunit.client.0.smithi102.stderr:53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.d 2024-08-19T14:19:12.194 INFO:tasks.workunit.client.0.smithi102.stderr:53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.194 INFO:tasks.workunit.client.0.smithi102.stderr:53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:12.194 INFO:tasks.workunit.client.0.smithi102.stderr:53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:12.194 INFO:tasks.workunit.client.0.smithi102.stderr:53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:12.194 INFO:tasks.workunit.client.0.smithi102.stderr:53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:12.194 INFO:tasks.workunit.client.0.smithi102.stderr:53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:12.194 INFO:tasks.workunit.client.0.smithi102.stderr:53838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.d last_scrub_stamp 2024-08-19T14:19:12.194 INFO:tasks.workunit.client.0.smithi102.stderr:53838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-08-19T14:19:12.194 INFO:tasks.workunit.client.0.smithi102.stderr:53838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.194 INFO:tasks.workunit.client.0.smithi102.stderr:53838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-08-19T14:19:12.194 INFO:tasks.workunit.client.0.smithi102.stderr:53838: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.195 INFO:tasks.workunit.client.0.smithi102.stderr:53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:19:09.230552+0000 '>' '' 2024-08-19T14:19:12.195 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:12.195 INFO:tasks.workunit.client.0.smithi102.stderr:53831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:12.195 INFO:tasks.workunit.client.0.smithi102.stderr:ub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:12.195 INFO:tasks.workunit.client.0.smithi102.stderr:53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:12.195 INFO:tasks.workunit.client.0.smithi102.stderr:53838: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:12.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:12.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:12.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53827 2024-08-19T14:19:12.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53835 2024-08-19T14:19:12.272 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.18 2024-08-19T14:19:12.273 INFO:tasks.workunit.client.0.smithi102.stderr:53956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.18 2024-08-19T14:19:12.273 INFO:tasks.workunit.client.0.smithi102.stderr:53956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-08-19T14:19:12.273 INFO:tasks.workunit.client.0.smithi102.stderr:53956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.273 INFO:tasks.workunit.client.0.smithi102.stderr:53956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-08-19T14:19:12.273 INFO:tasks.workunit.client.0.smithi102.stderr:53956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.273 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.273 INFO:tasks.workunit.client.0.smithi102.stderr:53956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.18 query 2024-08-19T14:19:12.273 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:12.273 INFO:tasks.workunit.client.0.smithi102.stderr:53956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:12.273 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:12.273 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:12.273 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.18 2024-08-19T14:19:12.273 INFO:tasks.workunit.client.0.smithi102.stderr:53956: instructing pg 1.18 on osd.5 to scrub 2024-08-19T14:19:12.274 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:12.274 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.18") | .state' 2024-08-19T14:19:12.274 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:12.274 INFO:tasks.workunit.client.0.smithi102.stderr:53956: dumped pgs 2024-08-19T14:19:12.274 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.18 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.274 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.18 2024-08-19T14:19:12.274 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.275 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:12.275 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:12.275 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:12.275 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:12.275 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:12.275 INFO:tasks.workunit.client.0.smithi102.stderr:53956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.18 last_scrub_stamp 2024-08-19T14:19:12.275 INFO:tasks.workunit.client.0.smithi102.stderr:53956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-08-19T14:19:12.275 INFO:tasks.workunit.client.0.smithi102.stderr:53956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.275 INFO:tasks.workunit.client.0.smithi102.stderr:53956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-08-19T14:19:12.275 INFO:tasks.workunit.client.0.smithi102.stderr:53956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.275 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:12.275 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:12.275 INFO:tasks.workunit.client.0.smithi102.stderr:53956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:12.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:12.288 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.17 2024-08-19T14:19:12.288 INFO:tasks.workunit.client.0.smithi102.stderr:53933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.17 2024-08-19T14:19:12.288 INFO:tasks.workunit.client.0.smithi102.stderr:53933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-08-19T14:19:12.288 INFO:tasks.workunit.client.0.smithi102.stderr:53933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.288 INFO:tasks.workunit.client.0.smithi102.stderr:53933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-08-19T14:19:12.289 INFO:tasks.workunit.client.0.smithi102.stderr:53933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.289 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.289 INFO:tasks.workunit.client.0.smithi102.stderr:53933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:12.289 INFO:tasks.workunit.client.0.smithi102.stderr:53933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.17 query 2024-08-19T14:19:12.289 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:12.289 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:12.289 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:12.289 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.17 2024-08-19T14:19:12.289 INFO:tasks.workunit.client.0.smithi102.stderr:53933: instructing pg 1.17 on osd.0 to scrub 2024-08-19T14:19:12.289 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.17") | .state' 2024-08-19T14:19:12.290 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:12.290 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:12.290 INFO:tasks.workunit.client.0.smithi102.stderr:53933: dumped pgs 2024-08-19T14:19:12.290 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.17 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.290 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.17 2024-08-19T14:19:12.290 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.290 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:12.290 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:12.290 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:12.290 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:12.290 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:53933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.17 last_scrub_stamp 2024-08-19T14:19:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:53933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-08-19T14:19:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:53933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:53933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:53933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-08-19T14:19:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:53933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:12.362 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.e 2024-08-19T14:19:12.362 INFO:tasks.workunit.client.0.smithi102.stderr:53845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.e 2024-08-19T14:19:12.362 INFO:tasks.workunit.client.0.smithi102.stderr:53845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-08-19T14:19:12.362 INFO:tasks.workunit.client.0.smithi102.stderr:53845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.362 INFO:tasks.workunit.client.0.smithi102.stderr:53845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.362 INFO:tasks.workunit.client.0.smithi102.stderr:53845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-08-19T14:19:12.363 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.363 INFO:tasks.workunit.client.0.smithi102.stderr:53845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-08-19T14:19:12.363 INFO:tasks.workunit.client.0.smithi102.stderr:53845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.e query 2024-08-19T14:19:12.363 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-08-19T14:19:12.363 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-08-19T14:19:12.363 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-08-19T14:19:12.363 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.e 2024-08-19T14:19:12.363 INFO:tasks.workunit.client.0.smithi102.stderr:53845: instructing pg 1.e on osd.1 to scrub 2024-08-19T14:19:12.363 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-08-19T14:19:12.363 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-08-19T14:19:12.363 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.e") | .state' 2024-08-19T14:19:12.363 INFO:tasks.workunit.client.0.smithi102.stderr:53845: dumped pgs 2024-08-19T14:19:12.363 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.e 2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.364 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.e 2024-08-19T14:19:12.364 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-08-19T14:17:40.848075+0000 2024-08-19T14:19:12.364 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-08-19T14:19:12.364 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-08-19T14:19:12.364 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-08-19T14:19:12.364 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-08-19T14:19:12.364 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-08-19T14:19:12.364 INFO:tasks.workunit.client.0.smithi102.stderr:53845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.e last_scrub_stamp 2024-08-19T14:19:12.364 INFO:tasks.workunit.client.0.smithi102.stderr:53845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-08-19T14:19:12.364 INFO:tasks.workunit.client.0.smithi102.stderr:53845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:19:12.364 INFO:tasks.workunit.client.0.smithi102.stderr:53845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-08-19T14:19:12.364 INFO:tasks.workunit.client.0.smithi102.stderr:53845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:19:12.364 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-08-19T14:17:40.848075+0000 '>' '' 2024-08-19T14:19:12.364 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-08-19T14:19:12.365 INFO:tasks.workunit.client.0.smithi102.stderr:53845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-08-19T14:19:12.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:19:12.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53839 2024-08-19T14:19:12.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53846 2024-08-19T14:19:12.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53856 2024-08-19T14:19:12.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53868 2024-08-19T14:19:12.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53871 2024-08-19T14:19:12.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53887 2024-08-19T14:19:12.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53899 2024-08-19T14:19:12.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53914 2024-08-19T14:19:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53927 2024-08-19T14:19:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53934 2024-08-19T14:19:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53948 2024-08-19T14:19:12.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 53979 2024-08-19T14:19:12.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 54004 2024-08-19T14:19:12.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 54020 2024-08-19T14:19:12.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 54027 2024-08-19T14:19:12.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 54065 2024-08-19T14:19:12.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-08-19T14:19:12.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 54072 2024-08-19T14:19:12.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-08-19T14:19:12.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-08-19T14:19:12.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-08-19T14:19:12.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: eval 'pids='\'''\''' 2024-08-19T14:19:12.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: pids= 2024-08-19T14:19:12.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:229: wait_background_check: return 0 2024-08-19T14:19:12.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:307: TEST_recovery_scrub_2: return_code=0 2024-08-19T14:19:12.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:308: TEST_recovery_scrub_2: '[' 0 -ne 0 ']' 2024-08-19T14:19:12.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:310: TEST_recovery_scrub_2: ERRORS=0 2024-08-19T14:19:12.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:311: TEST_recovery_scrub_2: test 2 -eq 0 2024-08-19T14:19:12.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: find td/osd-recovery-scrub 2024-08-19T14:19:12.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep '[^/]*\.pid' 2024-08-19T14:19:12.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: pidfile='td/osd-recovery-scrub/osd.1.pid 2024-08-19T14:19:12.372 INFO:tasks.workunit.client.0.smithi102.stderr:td/osd-recovery-scrub/mgr.x.pid 2024-08-19T14:19:12.372 INFO:tasks.workunit.client.0.smithi102.stderr:td/osd-recovery-scrub/osd.2.pid 2024-08-19T14:19:12.372 INFO:tasks.workunit.client.0.smithi102.stderr:td/osd-recovery-scrub/osd.5.pid 2024-08-19T14:19:12.372 INFO:tasks.workunit.client.0.smithi102.stderr:td/osd-recovery-scrub/osd.6.pid 2024-08-19T14:19:12.373 INFO:tasks.workunit.client.0.smithi102.stderr:td/osd-recovery-scrub/osd.0.pid 2024-08-19T14:19:12.373 INFO:tasks.workunit.client.0.smithi102.stderr:td/osd-recovery-scrub/osd.3.pid 2024-08-19T14:19:12.373 INFO:tasks.workunit.client.0.smithi102.stderr:td/osd-recovery-scrub/mon.a.pid 2024-08-19T14:19:12.373 INFO:tasks.workunit.client.0.smithi102.stderr:td/osd-recovery-scrub/osd.7.pid 2024-08-19T14:19:12.373 INFO:tasks.workunit.client.0.smithi102.stderr:td/osd-recovery-scrub/osd.4.pid' 2024-08-19T14:19:12.373 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: cat td/osd-recovery-scrub/osd.1.pid td/osd-recovery-scrub/mgr.x.pid td/osd-recovery-scrub/osd.2.pid td/osd-recovery-scrub/osd.5.pid td/osd-recovery-scrub/osd.6.pid td/osd-recovery-scrub/osd.0.pid td/osd-recovery-scrub/osd.3.pid td/osd-recovery-scrub/mon.a.pid td/osd-recovery-scrub/osd.7.pid td/osd-recovery-scrub/osd.4.pid 2024-08-19T14:19:12.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: pid='43191 2024-08-19T14:19:12.373 INFO:tasks.workunit.client.0.smithi102.stderr:40868 2024-08-19T14:19:12.373 INFO:tasks.workunit.client.0.smithi102.stderr:44567 2024-08-19T14:19:12.373 INFO:tasks.workunit.client.0.smithi102.stderr:48695 2024-08-19T14:19:12.373 INFO:tasks.workunit.client.0.smithi102.stderr:50071 2024-08-19T14:19:12.373 INFO:tasks.workunit.client.0.smithi102.stderr:41815 2024-08-19T14:19:12.373 INFO:tasks.workunit.client.0.smithi102.stderr:45943 2024-08-19T14:19:12.373 INFO:tasks.workunit.client.0.smithi102.stderr:40784 2024-08-19T14:19:12.374 INFO:tasks.workunit.client.0.smithi102.stderr:51447 2024-08-19T14:19:12.374 INFO:tasks.workunit.client.0.smithi102.stderr:47319' 2024-08-19T14:19:12.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:319: TEST_recovery_scrub_2: kill -0 43191 40868 44567 48695 50071 41815 45943 40784 51447 47319 2024-08-19T14:19:12.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:327: TEST_recovery_scrub_2: kill_daemons td/osd-recovery-scrub 2024-08-19T14:19:12.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:19:12.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:19:12.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:19:12.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:19:12.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:19:17.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:19:17.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:329: TEST_recovery_scrub_2: declare -a err_strings 2024-08-19T14:19:17.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:330: TEST_recovery_scrub_2: err_strings[0]='not scheduling scrubs due to active recovery' 2024-08-19T14:19:17.674 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: expr 8 - 1 2024-08-19T14:19:17.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: seq 0 7 2024-08-19T14:19:17.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:19:17.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2024-08-19T14:19:17.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:19:17.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2024-08-19T14:19:17.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:19:17.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2024-08-19T14:19:17.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:19:17.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2024-08-19T14:19:17.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:19:17.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.4.log 2024-08-19T14:19:17.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:19:17.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.5.log 2024-08-19T14:19:17.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:19:17.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.6.log 2024-08-19T14:19:17.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:19:17.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.7.log 2024-08-19T14:19:17.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:336: TEST_recovery_scrub_2: for err_string in "${err_strings[@]}" 2024-08-19T14:19:17.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:338: TEST_recovery_scrub_2: found=false 2024-08-19T14:19:17.708 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: expr 8 - 1 2024-08-19T14:19:17.709 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: seq 0 7 2024-08-19T14:19:17.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:19:17.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2024-08-19T14:19:17.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:19:17.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2024-08-19T14:19:17.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:19:17.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2024-08-19T14:19:17.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:19:17.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2024-08-19T14:19:17.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:19:17.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.4.log 2024-08-19T14:19:17.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:19:17.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.5.log 2024-08-19T14:19:17.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:19:17.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.6.log 2024-08-19T14:19:17.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T14:19:17.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.7.log 2024-08-19T14:19:17.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:346: TEST_recovery_scrub_2: '[' false = true ']' 2024-08-19T14:19:17.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:352: TEST_recovery_scrub_2: teardown td/osd-recovery-scrub 2024-08-19T14:19:17.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-08-19T14:19:17.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:19:17.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-08-19T14:19:17.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:19:17.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:19:17.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:19:17.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:19:17.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:19:17.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:19:17.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:19:17.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:19:17.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:19:17.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:19:17.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:19:17.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:19:17.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:19:17.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:19:17.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:19:17.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:19:17.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:19:17.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:19:17.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-08-19T14:19:17.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:19:17.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:19:17.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:19:17.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32181 2024-08-19T14:19:17.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:19:17.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:19:17.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:354: TEST_recovery_scrub_2: '[' 0 '!=' 0 ']' 2024-08-19T14:19:17.776 INFO:tasks.workunit.client.0.smithi102.stdout:TEST PASSED 2024-08-19T14:19:17.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:360: TEST_recovery_scrub_2: echo 'TEST PASSED' 2024-08-19T14:19:17.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:361: TEST_recovery_scrub_2: return 0 2024-08-19T14:19:17.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-08-19T14:19:17.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-recovery-scrub 0 2024-08-19T14:19:17.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-08-19T14:19:17.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-08-19T14:19:17.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-08-19T14:19:17.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:19:17.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:19:17.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:19:17.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:19:17.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:19:17.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:19:17.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:19:17.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:19:17.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:19:17.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:19:17.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:19:17.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:19:17.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:19:17.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:19:17.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:19:17.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:19:17.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:19:17.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-08-19T14:19:17.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-08-19T14:19:17.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:19:17.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:19:17.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32181 2024-08-19T14:19:17.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32181 2024-08-19T14:19:17.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:19:17.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:19:17.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-08-19T14:19:17.787 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-19T14:19:17.787 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-19T14:19:17.813 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2024-08-19T14:19:17.813 DEBUG:teuthology.orchestra.run.smithi102:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=78215f5a1cf5eaf321d506a66539942e12851f34 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2024-08-19T14:19:17.875 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-19T14:19:17.880 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-19T14:19:17.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-08-19T14:19:17.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-08-19T14:19:17.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-08-19T14:19:17.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-08-19T14:19:17.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-08-19T14:19:17.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-08-19T14:19:17.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-08-19T14:19:17.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-08-19T14:19:17.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-dump 2024-08-19T14:19:17.881 INFO:tasks.workunit.client.0.smithi102.stdout:This test is disabled 2024-08-19T14:19:17.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: echo 'This test is disabled' 2024-08-19T14:19:17.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: return 0 2024-08-19T14:19:17.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-08-19T14:19:17.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-dump 0 2024-08-19T14:19:17.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2024-08-19T14:19:17.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-08-19T14:19:17.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2024-08-19T14:19:17.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:19:17.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:19:17.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:19:17.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:19:17.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:19:17.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:19:17.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:19:17.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:19:17.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:19:17.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:19:17.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:19:17.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:19:17.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:19:17.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:19:17.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:19:17.885 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:19:17.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:19:17.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-08-19T14:19:17.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2024-08-19T14:19:17.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:19:17.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:19:17.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57225 2024-08-19T14:19:17.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57225 2024-08-19T14:19:17.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:19:17.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:19:17.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-08-19T14:19:17.889 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-19T14:19:17.890 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-19T14:19:17.954 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2024-08-19T14:19:17.954 DEBUG:teuthology.orchestra.run.smithi102:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=78215f5a1cf5eaf321d506a66539942e12851f34 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2024-08-19T14:19:18.015 INFO:tasks.workunit.client.0.smithi102.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-08-19T14:19:18.015 INFO:tasks.workunit.client.0.smithi102.stderr:++ TIMEOUT=300 2024-08-19T14:19:18.015 INFO:tasks.workunit.client.0.smithi102.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-08-19T14:19:18.015 INFO:tasks.workunit.client.0.smithi102.stderr:++ MAX_TIMEOUT=15 2024-08-19T14:19:18.015 INFO:tasks.workunit.client.0.smithi102.stderr:++ PG_NUM=4 2024-08-19T14:19:18.015 INFO:tasks.workunit.client.0.smithi102.stderr:++ TMPDIR=/tmp 2024-08-19T14:19:18.015 INFO:tasks.workunit.client.0.smithi102.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-08-19T14:19:18.015 INFO:tasks.workunit.client.0.smithi102.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-08-19T14:19:18.015 INFO:tasks.workunit.client.0.smithi102.stderr:++ type xmlstarlet 2024-08-19T14:19:18.015 INFO:tasks.workunit.client.0.smithi102.stderr:++ XMLSTARLET=xmlstarlet 2024-08-19T14:19:18.016 INFO:tasks.workunit.client.0.smithi102.stderr:+++ uname 2024-08-19T14:19:18.016 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' Linux = FreeBSD ']' 2024-08-19T14:19:18.016 INFO:tasks.workunit.client.0.smithi102.stderr:++ SED=sed 2024-08-19T14:19:18.016 INFO:tasks.workunit.client.0.smithi102.stderr:++ AWK=awk 2024-08-19T14:19:18.017 INFO:tasks.workunit.client.0.smithi102.stderr:+++ stty -a 2024-08-19T14:19:18.017 INFO:tasks.workunit.client.0.smithi102.stderr:+++ head -1 2024-08-19T14:19:18.017 INFO:tasks.workunit.client.0.smithi102.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-08-19T14:19:18.017 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-19T14:19:18.017 INFO:tasks.workunit.client.0.smithi102.stderr:++ termwidth= 2024-08-19T14:19:18.017 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-08-19T14:19:18.017 INFO:tasks.workunit.client.0.smithi102.stderr:++ DIFFCOLOPTS='-y ' 2024-08-19T14:19:18.017 INFO:tasks.workunit.client.0.smithi102.stderr:++ KERNCORE=kernel.core_pattern 2024-08-19T14:19:18.017 INFO:tasks.workunit.client.0.smithi102.stderr:++ EXTRA_OPTS= 2024-08-19T14:19:18.019 INFO:tasks.workunit.client.0.smithi102.stderr:++ test '' = TESTS 2024-08-19T14:19:18.020 INFO:tasks.workunit.client.0.smithi102.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2024-08-19T14:19:18.020 INFO:tasks.workunit.client.0.smithi102.stderr:++ uname 2024-08-19T14:19:18.021 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' Linux = FreeBSD ']' 2024-08-19T14:19:18.021 INFO:tasks.workunit.client.0.smithi102.stderr:+ use_ec_overwrite=true 2024-08-19T14:19:18.021 INFO:tasks.workunit.client.0.smithi102.stderr:+ getjson=no 2024-08-19T14:19:18.022 INFO:tasks.workunit.client.0.smithi102.stderr:+ jqfilter='def walk(f): 2024-08-19T14:19:18.022 INFO:tasks.workunit.client.0.smithi102.stderr: . as $in 2024-08-19T14:19:18.022 INFO:tasks.workunit.client.0.smithi102.stderr: | if type == "object" then 2024-08-19T14:19:18.022 INFO:tasks.workunit.client.0.smithi102.stderr: reduce keys[] as $key 2024-08-19T14:19:18.022 INFO:tasks.workunit.client.0.smithi102.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-08-19T14:19:18.022 INFO:tasks.workunit.client.0.smithi102.stderr: elif type == "array" then map( walk(f) ) | f 2024-08-19T14:19:18.022 INFO:tasks.workunit.client.0.smithi102.stderr: else f 2024-08-19T14:19:18.022 INFO:tasks.workunit.client.0.smithi102.stderr: end; 2024-08-19T14:19:18.022 INFO:tasks.workunit.client.0.smithi102.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-08-19T14:19:18.022 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-08-19T14:19:18.022 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-08-19T14:19:18.022 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.version) else . end) 2024-08-19T14:19:18.022 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-08-19T14:19:18.023 INFO:tasks.workunit.client.0.smithi102.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-08-19T14:19:18.027 INFO:tasks.workunit.client.0.smithi102.stderr:+ main osd-scrub-repair 2024-08-19T14:19:18.028 INFO:tasks.workunit.client.0.smithi102.stderr:+ local dir=td/osd-scrub-repair 2024-08-19T14:19:18.028 INFO:tasks.workunit.client.0.smithi102.stderr:+ shift 2024-08-19T14:19:18.028 INFO:tasks.workunit.client.0.smithi102.stderr:+ shopt -s -o xtrace 2024-08-19T14:19:18.028 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-19T14:19:18.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-08-19T14:19:18.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-08-19T14:19:18.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-08-19T14:19:18.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-08-19T14:19:18.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-08-19T14:19:18.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-08-19T14:19:18.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-08-19T14:19:18.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-08-19T14:19:18.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-repair 2024-08-19T14:19:18.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: local dir=td/osd-scrub-repair 2024-08-19T14:19:18.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:53: run: shift 2024-08-19T14:19:18.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_MON=127.0.0.1:7107 2024-08-19T14:19:18.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: CEPH_MON=127.0.0.1:7107 2024-08-19T14:19:18.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: export CEPH_ARGS 2024-08-19T14:19:18.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: uuidgen 2024-08-19T14:19:18.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none ' 2024-08-19T14:19:18.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2024-08-19T14:19:18.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:59: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2024-08-19T14:19:18.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-08-19T14:19:18.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: set 2024-08-19T14:19:18.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-08-19T14:19:18.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: local 'funcs=TEST_allow_repair_during_recovery 2024-08-19T14:19:18.032 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_auto_repair_bluestore_basic 2024-08-19T14:19:18.032 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_auto_repair_bluestore_failed 2024-08-19T14:19:18.032 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_auto_repair_bluestore_failed_norecov 2024-08-19T14:19:18.032 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_auto_repair_bluestore_scrub 2024-08-19T14:19:18.032 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_auto_repair_bluestore_tag 2024-08-19T14:19:18.032 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_auto_repair_erasure_coded_appends 2024-08-19T14:19:18.032 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_auto_repair_erasure_coded_overwrites 2024-08-19T14:19:18.032 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_corrupt_and_repair_jerasure_appends 2024-08-19T14:19:18.032 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2024-08-19T14:19:18.032 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_corrupt_and_repair_lrc_appends 2024-08-19T14:19:18.032 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_corrupt_and_repair_lrc_overwrites 2024-08-19T14:19:18.032 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_corrupt_and_repair_replicated 2024-08-19T14:19:18.032 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_corrupt_scrub_erasure_appends 2024-08-19T14:19:18.033 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_corrupt_scrub_erasure_overwrites 2024-08-19T14:19:18.033 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_corrupt_scrub_replicated 2024-08-19T14:19:18.033 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_corrupt_snapset_scrub_rep 2024-08-19T14:19:18.033 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_list_missing_erasure_coded_appends 2024-08-19T14:19:18.033 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_list_missing_erasure_coded_overwrites 2024-08-19T14:19:18.033 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_periodic_scrub_replicated 2024-08-19T14:19:18.033 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_repair_stats 2024-08-19T14:19:18.033 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_repair_stats_ec 2024-08-19T14:19:18.033 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_request_scrub_priority 2024-08-19T14:19:18.033 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_scrub_warning 2024-08-19T14:19:18.033 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_skip_non_repair_during_recovery 2024-08-19T14:19:18.033 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_unfound_erasure_coded_appends 2024-08-19T14:19:18.033 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_unfound_erasure_coded_overwrites' 2024-08-19T14:19:18.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T14:19:18.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T14:19:18.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T14:19:18.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T14:19:18.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:19:18.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:19:18.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:19:18.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:19:18.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:19:18.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:19:18.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:19:18.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:19:18.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:19:18.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:19:18.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:19:18.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:19:18.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:19:18.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:19:18.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:19:18.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:19:18.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:19:18.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:19:18.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:19:18.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:19:18.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:19:18.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:19:18.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:19:18.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:19:18.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:19:18.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:19:18.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:19:18.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:19:18.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T14:19:18.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T14:19:18.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:19:18.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:19:18.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T14:19:18.044 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T14:19:18.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-19T14:19:18.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-19T14:19:18.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T14:19:18.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T14:19:18.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2024-08-19T14:19:18.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:114: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2024-08-19T14:19:18.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_repair_during_recovery: local poolname=rbd 2024-08-19T14:19:18.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:117: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-08-19T14:19:18.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T14:19:18.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T14:19:18.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T14:19:18.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T14:19:18.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T14:19:18.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-08-19T14:19:18.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T14:19:18.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:19:18.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:19:18.402 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:19:18.403 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:19:18.403 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:19:18.403 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:19:18.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-08-19T14:19:18.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T14:19:18.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T14:19:18.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:19:18.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:19:18.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T14:19:18.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T14:19:18.587 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:19:18.587 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:19:18.587 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:19:18.588 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:19:18.588 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:19:18.588 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:19:18.588 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:19:18.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:19:18.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T14:19:18.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T14:19:18.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:19:18.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:19:18.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T14:19:18.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T14:19:18.670 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:19:18.670 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:19:18.670 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:19:18.670 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:19:18.670 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:19:18.671 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:19:18.671 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:19:18.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:19:18.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T14:19:18.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:118: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-08-19T14:19:18.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T14:19:18.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T14:19:18.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T14:19:18.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T14:19:18.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T14:19:18.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T14:19:19.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T14:19:19.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:19:19.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:19:19.555 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:19:19.556 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:19:19.556 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:19:19.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:19:19.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:19:19.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:19:19.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:119: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-08-19T14:19:19.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:19:19.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:19:19.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T14:19:19.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:19:19.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:19:19.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:19:19.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:19:19.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:19:19.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:19:19.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:19:19.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:19:19.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:19:19.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:19:19.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:19:19.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:19:19.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:19:19.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:19:19.587 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:19:19.588 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:19:19.588 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:19:19.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:19:19.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:19:19.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:19:19.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:19:19.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:19:19.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:19:19.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:19:19.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:19:19.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:19:19.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:19:19.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:19:19.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:19:19.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-08-19T14:19:19.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:19:19.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:19:19.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=15682ade-0286-4ec5-bc37-327d53be7be8 2024-08-19T14:19:19.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 15682ade-0286-4ec5-bc37-327d53be7be8' 2024-08-19T14:19:19.590 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 15682ade-0286-4ec5-bc37-327d53be7be8 2024-08-19T14:19:19.590 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:19:19.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBnVMNma5wKJBAAzdq3EHBHg0n4tHHZGdhjyw== 2024-08-19T14:19:19.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBnVMNma5wKJBAAzdq3EHBHg0n4tHHZGdhjyw=="}' 2024-08-19T14:19:19.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 15682ade-0286-4ec5-bc37-327d53be7be8 -i td/osd-scrub-repair/0/new.json 2024-08-19T14:19:19.802 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:19:19.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T14:19:19.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQBnVMNma5wKJBAAzdq3EHBHg0n4tHHZGdhjyw== --osd-uuid 15682ade-0286-4ec5-bc37-327d53be7be8 2024-08-19T14:19:19.843 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:19.843+0000 7feea7032640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:19:19.853 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:19.853+0000 7feea7032640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:19:19.855 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:19.854+0000 7feea7032640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:19:19.855 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:19.855+0000 7feea7032640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:19:19.855 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:19.855+0000 7feea7032640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:19:19.856 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:19.855+0000 7feea7032640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:19:19.945 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:19.944+0000 7feea7032640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T14:19:25.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T14:19:25.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:19:25.407 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T14:19:25.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T14:19:25.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:19:25.758 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T14:19:25.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T14:19:25.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-08-19T14:19:25.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:19:25.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:19:25.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:19:25.787 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:25.787+0000 7fe4cb241640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:19:25.789 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:25.789+0000 7fe4cb241640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:19:25.790 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:25.790+0000 7fe4cb241640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:19:26.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T14:19:26.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:19:26.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:19:26.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:19:26.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:19:26.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:26.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:19:26.024 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:19:26.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:26.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:19:26.304 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:26.304+0000 7fe4cb241640 -1 Falling back to public interface 2024-08-19T14:19:26.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:19:27.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:19:27.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:27.332 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:19:27.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:19:27.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:27.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:19:27.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:19:28.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:19:28.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:28.606 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:19:28.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:19:28.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:28.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:19:28.626 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:28.626+0000 7fe4cb241640 -1 osd.0 0 log_to_monitors true 2024-08-19T14:19:28.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:19:29.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:19:29.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:29.919 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:19:29.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:19:29.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:29.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:19:30.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:19:31.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:19:31.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:31.199 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:19:31.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:19:31.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:31.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:19:31.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:19:32.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:19:32.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:32.569 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:19:32.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:19:32.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:32.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:19:32.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:19:33.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:19:33.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:33.844 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:19:33.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:19:33.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:33.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:19:34.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:19:35.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:19:35.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:35.123 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:19:35.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:19:35.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:35.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:19:35.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:19:36.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:19:36.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:36.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:19:36.401 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:19:36.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:36.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:19:36.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:19:37.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:19:37.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:37.679 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:19:37.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:19:37.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:37.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:19:37.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:19:38.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:19:38.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:38.959 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:19:38.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:19:38.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:38.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:19:39.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:19:40.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:19:40.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:40.394 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:19:40.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:19:40.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:40.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:19:40.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:19:41.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:19:41.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:41.846 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:19:41.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:19:41.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:41.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:19:42.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:19:43.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:19:43.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:43.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:19:43.117 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:19:43.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:43.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:19:43.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:19:44.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:19:44.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:44.421 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:19:44.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:19:44.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:44.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:19:44.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:19:45.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:19:45.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:45.794 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:19:45.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:19:45.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:45.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:19:46.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:19:47.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:19:47.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:47.406 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:19:47.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:19:47.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:47.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:19:47.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:19:48.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:19:48.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:48.782 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:19:48.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:19:48.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:48.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:19:49.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:19:50.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:19:50.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:50.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:19:50.246 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:19:50.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:50.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:19:50.515 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3383386868,v1:127.0.0.1:6803/3383386868] [v2:127.0.0.1:6804/3383386868,v1:127.0.0.1:6805/3383386868] exists,up 15682ade-0286-4ec5-bc37-327d53be7be8 2024-08-19T14:19:50.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:19:50.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:19:50.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:19:50.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:122: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-08-19T14:19:50.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:19:50.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:19:50.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T14:19:50.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:19:50.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:19:50.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:19:50.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:19:50.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:19:50.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:19:50.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:19:50.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:19:50.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:19:50.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:19:50.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:19:50.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:19:50.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:19:50.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:19:50.521 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:19:50.521 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:19:50.521 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:19:50.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:19:50.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:19:50.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:19:50.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:19:50.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:19:50.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:19:50.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:19:50.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:19:50.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:19:50.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:19:50.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:19:50.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:19:50.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-08-19T14:19:50.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:19:50.522 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:19:50.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3c207105-bff7-4f0f-b2b8-d8dfa2e013ed 2024-08-19T14:19:50.522 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 3c207105-bff7-4f0f-b2b8-d8dfa2e013ed 2024-08-19T14:19:50.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3c207105-bff7-4f0f-b2b8-d8dfa2e013ed' 2024-08-19T14:19:50.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:19:50.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCGVMNmj5kNIBAAFdQnRwxZERSeyaOPTgo83A== 2024-08-19T14:19:50.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCGVMNmj5kNIBAAFdQnRwxZERSeyaOPTgo83A=="}' 2024-08-19T14:19:50.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3c207105-bff7-4f0f-b2b8-d8dfa2e013ed -i td/osd-scrub-repair/1/new.json 2024-08-19T14:19:50.886 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:19:50.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T14:19:50.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQCGVMNmj5kNIBAAFdQnRwxZERSeyaOPTgo83A== --osd-uuid 3c207105-bff7-4f0f-b2b8-d8dfa2e013ed 2024-08-19T14:19:50.928 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:50.928+0000 7f42a660d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:19:50.971 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:50.969+0000 7f42a660d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:19:50.972 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:50.970+0000 7f42a660d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:19:50.972 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:50.971+0000 7f42a660d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:19:50.972 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:50.971+0000 7f42a660d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:19:50.972 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:50.971+0000 7f42a660d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:19:51.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:51.036+0000 7f42a660d640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T14:19:56.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T14:19:56.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:19:56.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T14:19:56.885 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T14:19:56.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:19:57.231 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T14:19:57.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T14:19:57.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-08-19T14:19:57.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:19:57.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:19:57.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:19:57.260 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:57.260+0000 7f4cee42c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:19:57.262 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:57.262+0000 7f4cee42c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:19:57.263 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:57.263+0000 7f4cee42c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:19:57.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T14:19:57.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:19:57.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:19:57.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:19:57.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:19:57.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:57.500 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:19:57.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:19:57.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:57.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:19:57.783 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:19:57.783+0000 7f4cee42c640 -1 Falling back to public interface 2024-08-19T14:19:57.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:19:58.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:19:58.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:19:58.791 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:19:58.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:19:58.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:19:58.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:19:59.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:00.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:00.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:00.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:20:00.063 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:20:00.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:00.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:00.186 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:20:00.185+0000 7f4cee42c640 -1 osd.1 0 log_to_monitors true 2024-08-19T14:20:00.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:01.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:01.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:01.396 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:20:01.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:20:01.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:01.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:01.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:02.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:02.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:02.671 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:20:02.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:20:02.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:02.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:03.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:04.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:04.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:04.011 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:20:04.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:20:04.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:04.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:04.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:05.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:05.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:05.285 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:20:05.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:20:05.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:05.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:05.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:06.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:06.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:06.560 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:20:06.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:20:06.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:06.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:06.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:07.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:07.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:07.839 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:20:07.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:20:07.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:07.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:08.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:09.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:09.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:09.134 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:20:09.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:20:09.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:09.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:09.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:10.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:10.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:10.411 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:20:10.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:20:10.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:10.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:10.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:11.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:11.688 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:20:11.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:11.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:20:11.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:11.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:11.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:12.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:12.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:12.973 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:20:12.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:20:12.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:12.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:13.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:14.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:14.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:14.757 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:20:14.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:20:14.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:14.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:15.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:16.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:16.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:16.105 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:20:16.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:20:16.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:16.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:16.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:17.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:17.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:17.601 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:20:17.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:20:17.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:17.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:17.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:18.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:18.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:18.961 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:20:18.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:20:18.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:18.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:19.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:20.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:20.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:20.514 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:20:20.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:20:20.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:20.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:20.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:21.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:21.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:21.788 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:20:21.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:20:21.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:21.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:22.147 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1751523816,v1:127.0.0.1:6811/1751523816] [v2:127.0.0.1:6812/1751523816,v1:127.0.0.1:6813/1751523816] exists,up 3c207105-bff7-4f0f-b2b8-d8dfa2e013ed 2024-08-19T14:20:22.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:20:22.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:20:22.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:20:22.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_repair_during_recovery: create_rbd_pool 2024-08-19T14:20:22.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T14:20:22.404 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-08-19T14:20:22.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T14:20:22.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T14:20:22.948 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' created 2024-08-19T14:20:22.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T14:20:23.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T14:20:25.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_repair_during_recovery: wait_for_clean 2024-08-19T14:20:25.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:20:25.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:20:25.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:20:25.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:20:25.650 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:20:25.650 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:20:25.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:20:25.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:20:25.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:20:25.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:20:25.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:20:25.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:20:25.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:20:25.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:20:25.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:20:26.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:20:26.011 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T14:20:26.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:20:26.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:20:26.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:20:26.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836491 2024-08-19T14:20:26.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836491 2024-08-19T14:20:26.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491' 2024-08-19T14:20:26.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:20:26.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:20:26.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-08-19T14:20:26.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-08-19T14:20:26.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491 1-42949672967' 2024-08-19T14:20:26.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:20:26.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836491 2024-08-19T14:20:26.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:20:26.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:20:26.227 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836491 2024-08-19T14:20:26.227 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:20:26.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836491 2024-08-19T14:20:26.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-08-19T14:20:26.229 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836491 2024-08-19T14:20:26.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:20:26.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836491 2024-08-19T14:20:26.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:20:27.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:20:27.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:20:27.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836491 2024-08-19T14:20:27.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:20:27.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-08-19T14:20:27.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:20:27.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:20:27.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-08-19T14:20:27.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:20:27.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-08-19T14:20:27.789 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672967 2024-08-19T14:20:27.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-08-19T14:20:27.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:20:28.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-08-19T14:20:28.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:20:28.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:20:28.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:20:28.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T14:20:28.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:20:28.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:20:28.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:20:28.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:20:28.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:20:28.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:20:28.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:20:28.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:20:28.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:20:28.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:20:28.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:20:29.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T14:20:29.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:20:29.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:20:29.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-08-19T14:20:29.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T14:20:29.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-08-19T14:20:29.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-08-19T14:20:29.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T14:20:29.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T14:20:29.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T14:20:29.587 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T14:20:29.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T14:20:29.980 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T14:20:29.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T14:20:29.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T14:20:29.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-08-19T14:20:30.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2024-08-19T14:20:30.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-08-19T14:20:30.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-08-19T14:20:30.245 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-08-19T14:20:30.245 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-08-19T14:20:30.245 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:20:30.245 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-08-19T14:20:30.245 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:20:30.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T14:20:30.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-08-19T14:20:30.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T14:20:30.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-08-19T14:20:30.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-08-19T14:20:30.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-08-19T14:20:30.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-08-19T14:20:30.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T14:20:30.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:20:30.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:20:30.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T14:20:30.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:20:30.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T14:20:30.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:20:30.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:20:30.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T14:20:30.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:20:30.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T14:20:30.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:20:30.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:20:30.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:20:30.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:20:30.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:20:31.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:20:31.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T14:20:31.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:20:31.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:20:31.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T14:20:31.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:20:31.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:20:31.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-08-19T14:20:32.346 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-08-19T14:20:32.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T14:20:32.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:20:32.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:20:32.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T14:20:32.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:20:32.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:20:32.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:20:32.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:20:32.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:20:32.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:20:32.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:20:32.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:20:32.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:20:32.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:20:32.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:20:32.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:20:32.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:20:32.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:20:32.869 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:20:32.869 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:20:32.869 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:20:32.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:20:32.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:20:32.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:20:32.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:20:32.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:20:32.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:20:32.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:20:32.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:20:32.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:20:32.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:20:32.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:20:32.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:20:32.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T14:20:32.872 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T14:20:32.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:20:32.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T14:20:32.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T14:20:32.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:20:32.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:20:32.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:20:32.898 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:20:32.898+0000 7f2698718640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:20:32.901 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:20:32.900+0000 7f2698718640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:20:32.902 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:20:32.902+0000 7f2698718640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:20:33.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T14:20:33.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:20:33.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:20:33.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:20:33.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:20:33.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:33.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:20:33.146 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:20:33.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:33.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:20:33.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:33.423 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:20:33.423+0000 7f2698718640 -1 Falling back to public interface 2024-08-19T14:20:34.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:34.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:34.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:20:34.413 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:20:34.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:34.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:20:34.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:35.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:35.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:35.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:20:35.687 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:20:35.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:35.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:20:35.911 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:20:35.910+0000 7f2698718640 -1 osd.0 22 log_to_monitors true 2024-08-19T14:20:35.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:36.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:36.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:36.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:20:36.960 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:20:36.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:36.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:20:37.385 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/2231499159,v1:127.0.0.1:6803/2231499159] [v2:127.0.0.1:6804/2231499159,v1:127.0.0.1:6805/2231499159] exists,up 15682ade-0286-4ec5-bc37-327d53be7be8 2024-08-19T14:20:37.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:20:37.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:20:37.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:20:37.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:20:37.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:20:37.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:20:37.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:20:37.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:20:37.386 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:20:37.386 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:20:37.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:20:37.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:20:37.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:20:37.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:20:37.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:20:37.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:20:37.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:20:37.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:20:37.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:20:37.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:20:37.789 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T14:20:37.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:20:37.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:20:37.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:20:37.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149698 2024-08-19T14:20:37.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149698 2024-08-19T14:20:37.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698' 2024-08-19T14:20:37.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:20:37.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:20:38.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-08-19T14:20:38.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-08-19T14:20:38.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698 1-42949672971' 2024-08-19T14:20:38.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:20:38.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149698 2024-08-19T14:20:38.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:20:38.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:20:38.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149698 2024-08-19T14:20:38.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:20:38.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149698 2024-08-19T14:20:38.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-08-19T14:20:38.007 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 111669149698 2024-08-19T14:20:38.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:20:38.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149698 2024-08-19T14:20:38.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:20:39.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:20:39.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:20:39.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149698 2024-08-19T14:20:39.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:20:40.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T14:20:40.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:20:40.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149698 -lt 111669149698 2024-08-19T14:20:40.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:20:40.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-08-19T14:20:40.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:20:40.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:20:40.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-08-19T14:20:40.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:20:40.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-08-19T14:20:40.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-08-19T14:20:40.823 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949672971 2024-08-19T14:20:40.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:20:41.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672971 2024-08-19T14:20:41.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:20:41.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:20:41.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:20:41.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T14:20:41.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:20:41.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:20:41.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:20:41.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:20:41.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:20:41.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:20:41.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:20:41.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:20:41.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:20:41.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:20:41.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:20:42.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T14:20:42.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:20:42.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:20:42.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-08-19T14:20:42.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-08-19T14:20:42.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T14:20:42.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-08-19T14:20:42.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:20:42.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-08-19T14:20:42.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-08-19T14:20:42.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-08-19T14:20:42.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-08-19T14:20:42.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T14:20:42.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:20:42.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:20:42.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T14:20:42.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T14:20:22.831566+0000 2024-08-19T14:20:42.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-08-19T14:20:42.854 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 1.3 on osd.1 to repair 2024-08-19T14:20:42.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-08-19T14:20:22.831566+0000 2024-08-19T14:20:42.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-08-19T14:20:42.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:20:22.831566+0000 2024-08-19T14:20:42.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:20:42.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:20:42.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:20:42.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T14:20:42.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T14:20:42.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:20:42.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:20:42.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T14:20:43.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:20:22.831566+0000 '>' 2024-08-19T14:20:22.831566+0000 2024-08-19T14:20:43.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:20:44.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:20:44.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:20:44.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T14:20:44.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T14:20:44.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:20:44.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:20:44.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T14:20:44.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:20:22.831566+0000 '>' 2024-08-19T14:20:22.831566+0000 2024-08-19T14:20:44.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:20:45.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:20:45.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:20:45.403 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T14:20:45.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T14:20:45.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:20:45.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:20:45.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T14:20:45.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:20:22.831566+0000 '>' 2024-08-19T14:20:22.831566+0000 2024-08-19T14:20:45.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:20:46.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:20:46.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:20:46.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T14:20:46.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T14:20:46.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:20:46.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:20:46.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T14:20:46.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:20:22.831566+0000 '>' 2024-08-19T14:20:22.831566+0000 2024-08-19T14:20:46.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:20:47.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:20:47.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:20:47.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T14:20:47.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T14:20:47.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:20:47.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:20:47.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T14:20:48.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:20:22.831566+0000 '>' 2024-08-19T14:20:22.831566+0000 2024-08-19T14:20:48.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:20:49.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:20:49.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:20:49.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T14:20:49.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T14:20:49.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:20:49.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:20:49.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T14:20:49.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:20:22.831566+0000 '>' 2024-08-19T14:20:22.831566+0000 2024-08-19T14:20:49.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:20:50.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:20:50.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:20:50.497 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T14:20:50.497 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T14:20:50.497 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:20:50.497 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:20:50.497 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T14:20:50.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:20:43.213461+0000 '>' 2024-08-19T14:20:22.831566+0000 2024-08-19T14:20:50.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:20:50.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:20:50.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:20:50.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:20:50.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T14:20:50.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:20:50.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:20:50.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:20:50.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:20:50.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T14:20:50.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:20:50.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T14:20:50.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:20:50.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:20:50.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:20:50.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:20:50.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:20:51.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:20:51.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:20:51.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:20:51.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:20:51.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T14:20:51.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:20:51.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:20:51.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-08-19T14:20:52.218 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-08-19T14:20:52.218 INFO:tasks.workunit.client.0.smithi102.stdout:_ 2024-08-19T14:20:52.218 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 2024-08-19T14:20:52.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T14:20:52.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:20:52.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:20:52.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:20:52.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:20:52.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:20:52.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:20:52.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:20:52.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:20:52.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:20:52.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:20:52.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:20:52.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:20:52.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:20:52.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:20:52.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:20:52.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:20:52.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:20:52.488 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:20:52.488 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:20:52.488 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:20:52.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:20:52.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:20:52.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:20:52.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:20:52.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:20:52.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:20:52.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:20:52.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:20:52.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:20:52.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:20:52.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:20:52.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:20:52.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:20:52.490 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T14:20:52.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:20:52.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T14:20:52.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:20:52.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:20:52.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:20:52.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:20:52.516 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:20:52.516+0000 7fbb0781f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:20:52.518 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:20:52.518+0000 7fbb0781f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:20:52.519 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:20:52.519+0000 7fbb0781f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:20:52.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:20:52.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:20:52.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:20:52.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:20:52.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:20:52.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:52.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:20:52.761 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:20:52.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:52.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:53.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:53.043 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:20:53.043+0000 7fbb0781f640 -1 Falling back to public interface 2024-08-19T14:20:54.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:54.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:54.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:20:54.028 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:20:54.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:54.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:54.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:55.229 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:20:55.229+0000 7fbb0781f640 -1 osd.1 28 log_to_monitors true 2024-08-19T14:20:55.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:55.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:55.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:20:55.296 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:20:55.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:55.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:55.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:20:56.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:20:56.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:20:56.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:20:56.580 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:20:56.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:20:56.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:20:56.858 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/2633774974,v1:127.0.0.1:6811/2633774974] [v2:127.0.0.1:6812/2633774974,v1:127.0.0.1:6813/2633774974] exists,up 3c207105-bff7-4f0f-b2b8-d8dfa2e013ed 2024-08-19T14:20:56.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:20:56.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:20:56.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:20:56.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:20:56.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:20:56.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:20:56.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:20:56.858 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:20:56.859 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:20:56.859 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:20:56.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:20:56.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:20:56.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:20:56.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:20:56.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:20:56.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:20:56.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:20:56.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:20:56.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:20:57.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:20:57.207 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T14:20:57.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:20:57.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:20:57.207 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:20:57.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149703 2024-08-19T14:20:57.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149703 2024-08-19T14:20:57.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149703' 2024-08-19T14:20:57.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:20:57.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:20:57.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-08-19T14:20:57.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-08-19T14:20:57.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149703 1-137438953474' 2024-08-19T14:20:57.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:20:57.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149703 2024-08-19T14:20:57.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:20:57.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:20:57.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149703 2024-08-19T14:20:57.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:20:57.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149703 2024-08-19T14:20:57.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149703' 2024-08-19T14:20:57.417 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 111669149703 2024-08-19T14:20:57.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:20:57.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149703 -lt 111669149703 2024-08-19T14:20:57.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:20:57.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-137438953474 2024-08-19T14:20:57.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:20:57.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:20:57.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-137438953474 2024-08-19T14:20:57.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:20:57.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-08-19T14:20:57.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-08-19T14:20:57.691 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 137438953474 2024-08-19T14:20:57.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:20:57.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953474 -lt 137438953474 2024-08-19T14:20:57.962 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:20:57.962 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:20:57.962 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:20:58.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T14:20:58.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:20:58.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:20:58.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:20:58.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:20:58.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:20:58.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:20:58.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:20:58.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:20:58.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:20:58.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:20:58.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:20:58.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T14:20:58.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:20:58.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:20:58.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T14:20:58.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T14:20:58.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T14:20:58.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:20:58.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:20:58.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:20:58.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:20:58.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:20:58.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:20:58.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:20:58.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:20:59.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:20:59.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:20:59.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:20:59.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:20:59.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:20:59.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:20:59.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:20:59.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:20:59.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:20:59.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:20:59.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:20:59.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:20:59.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:20:59.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:20:59.095 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:20:59.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:20:59.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:20:59.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:20:59.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:20:59.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:20:59.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T14:20:59.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T14:20:59.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T14:20:59.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T14:20:59.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:20:59.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:20:59.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:20:59.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:20:59.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:20:59.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:20:59.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:20:59.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:20:59.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:20:59.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:20:59.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:20:59.100 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:20:59.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:20:59.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:20:59.101 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:20:59.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:20:59.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:20:59.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:20:59.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:20:59.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:20:59.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:20:59.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:20:59.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:20:59.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:20:59.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:20:59.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:20:59.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:20:59.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:20:59.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T14:20:59.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T14:20:59.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:20:59.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:20:59.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T14:20:59.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T14:20:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T14:20:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T14:20:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T14:20:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2024-08-19T14:20:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2024-08-19T14:20:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='1' ['pool_name']='testpool' ['extras']=' --osd_scrub_auto_repair=true') 2024-08-19T14:20:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: local -A cluster_conf 2024-08-19T14:20:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:445: TEST_auto_repair_bluestore_basic: local extr_dbg=3 2024-08-19T14:20:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:446: TEST_auto_repair_bluestore_basic: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2024-08-19T14:20:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-scrub-repair 2024-08-19T14:20:59.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-08-19T14:20:59.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-08-19T14:20:59.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=1 2024-08-19T14:20:59.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=testpool 2024-08-19T14:20:59.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=testpool 2024-08-19T14:20:59.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local 'extra_pars= --osd_scrub_auto_repair=true' 2024-08-19T14:20:59.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-08-19T14:20:59.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-08-19T14:20:59.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-08-19T14:21:00.674 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 432c1b07-ec10-4a79-9447-2bba7c043e8e 2024-08-19T14:21:00.896 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:21:00.940 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:00.940+0000 7fcc323bc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:21:00.947 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:00.946+0000 7fcc323bc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:21:00.948 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:00.948+0000 7fcc323bc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:21:00.949 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:00.948+0000 7fcc323bc640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:21:00.949 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:00.948+0000 7fcc323bc640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:21:00.949 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:00.948+0000 7fcc323bc640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:21:01.030 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:01.030+0000 7fcc323bc640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T14:21:06.496 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T14:21:06.846 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T14:21:06.878 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:06.877+0000 7fabfadc2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:21:06.923 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:06.922+0000 7fabfadc2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:21:06.923 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:06.923+0000 7fabfadc2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:21:07.108 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:21:07.441 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:07.441+0000 7fabfadc2640 -1 Falling back to public interface 2024-08-19T14:21:08.372 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:21:09.646 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:21:09.729 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:09.728+0000 7fabfadc2640 -1 osd.0 0 log_to_monitors true 2024-08-19T14:21:10.988 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:21:12.268 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:21:13.647 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:21:14.920 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:21:16.196 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:21:17.472 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:21:18.747 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:21:20.025 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:21:21.315 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:21:22.742 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:21:24.122 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:21:25.871 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:21:27.526 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:21:29.050 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:21:30.714 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:21:31.006 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3087064823,v1:127.0.0.1:6803/3087064823] [v2:127.0.0.1:6804/3087064823,v1:127.0.0.1:6805/3087064823] exists,up 432c1b07-ec10-4a79-9447-2bba7c043e8e 2024-08-19T14:21:31.009 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 5dd314c5-4c25-471c-b41b-b281a95a06c9 2024-08-19T14:21:31.374 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:21:31.419 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:31.418+0000 7f266d215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:21:31.461 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:31.460+0000 7f266d215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:21:31.462 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:31.461+0000 7f266d215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:21:31.462 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:31.462+0000 7f266d215640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:21:31.462 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:31.462+0000 7f266d215640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:21:31.463 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:31.462+0000 7f266d215640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:21:31.525 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:31.525+0000 7f266d215640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T14:21:37.221 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T14:21:37.562 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T14:21:37.594 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:37.593+0000 7fd4ee12a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:21:37.596 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:37.595+0000 7fd4ee12a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:21:37.597 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:37.597+0000 7fd4ee12a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:21:37.828 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:21:38.119 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:38.119+0000 7fd4ee12a640 -1 Falling back to public interface 2024-08-19T14:21:39.095 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:21:40.371 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:21:40.434 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:21:40.433+0000 7fd4ee12a640 -1 osd.1 0 log_to_monitors true 2024-08-19T14:21:41.647 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:21:42.926 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:21:44.308 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:21:45.587 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:21:46.864 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:21:48.191 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:21:49.469 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:21:50.750 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:21:52.043 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:21:53.639 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:21:55.040 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:21:56.408 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:21:57.869 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:21:59.383 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:22:00.943 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:22:02.385 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:22:02.723 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4183509402,v1:127.0.0.1:6811/4183509402] [v2:127.0.0.1:6812/4183509402,v1:127.0.0.1:6813/4183509402] exists,up 5dd314c5-4c25-471c-b41b-b281a95a06c9 2024-08-19T14:22:02.727 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 65f0c93c-0f71-4b49-a297-9a286fc55a45 2024-08-19T14:22:03.074 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:22:03.118 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:22:03.117+0000 7f71c6e82640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:22:03.140 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:22:03.139+0000 7f71c6e82640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:22:03.141 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:22:03.140+0000 7f71c6e82640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:22:03.141 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:22:03.141+0000 7f71c6e82640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:22:03.141 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:22:03.141+0000 7f71c6e82640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:22:03.141 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:22:03.141+0000 7f71c6e82640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:22:03.215 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:22:03.215+0000 7f71c6e82640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-08-19T14:22:08.880 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T14:22:09.222 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T14:22:09.254 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:22:09.253+0000 7f510da1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:22:09.256 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:22:09.255+0000 7f510da1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:22:09.257 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:22:09.256+0000 7f510da1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:22:09.488 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:22:09.782 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:22:09.782+0000 7f510da1c640 -1 Falling back to public interface 2024-08-19T14:22:10.756 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:22:12.047 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:22:12.156 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:22:12.155+0000 7f510da1c640 -1 osd.2 0 log_to_monitors true 2024-08-19T14:22:13.382 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:22:14.655 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:22:16.030 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:22:17.304 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:22:18.580 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:22:19.888 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:22:21.167 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:22:22.441 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:22:23.718 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:22:25.355 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:22:26.937 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:22:28.346 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:22:29.620 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:22:31.239 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:22:32.526 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:22:33.923 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:22:34.194 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3716392706,v1:127.0.0.1:6819/3716392706] [v2:127.0.0.1:6820/3716392706,v1:127.0.0.1:6821/3716392706] exists,up 65f0c93c-0f71-4b49-a297-9a286fc55a45 2024-08-19T14:22:34.567 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'testpool' created 2024-08-19T14:22:36.253 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836497 2024-08-19T14:22:37.797 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672972 2024-08-19T14:22:38.067 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509447 2024-08-19T14:22:40.533 INFO:tasks.workunit.client.0.smithi102.stdout:standard_scrub_cluster: dbg: test pool is testpool 1 2024-08-19T14:22:40.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: local poolid=1 2024-08-19T14:22:40.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_basic: local poolname=testpool 2024-08-19T14:22:40.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:450: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2024-08-19T14:22:41.010 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 2 2024-08-19T14:22:41.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-08-19T14:22:41.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:22:41.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:22:41.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:22:41.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:22:41.023 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:22:41.023 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:22:41.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:22:41.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:22:41.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:22:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:22:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:22:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:22:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:22:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:22:41.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:22:41.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:22:41.400 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:22:41.400 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:22:41.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:22:41.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:22:41.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:22:41.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-08-19T14:22:41.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-08-19T14:22:41.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-08-19T14:22:41.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:22:41.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:22:41.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-08-19T14:22:41.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-08-19T14:22:41.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672974' 2024-08-19T14:22:41.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:22:41.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:22:41.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-08-19T14:22:41.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-08-19T14:22:41.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672974 2-64424509449' 2024-08-19T14:22:41.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:22:41.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-08-19T14:22:41.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:22:41.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:22:41.772 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-08-19T14:22:41.772 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:22:41.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-08-19T14:22:41.773 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836500 2024-08-19T14:22:41.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-08-19T14:22:41.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:22:42.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836500 2024-08-19T14:22:42.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:22:43.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:22:43.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:22:43.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-08-19T14:22:43.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:22:43.319 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-08-19T14:22:43.319 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:22:43.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:22:43.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-08-19T14:22:43.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:22:43.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-08-19T14:22:43.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-08-19T14:22:43.322 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672974 2024-08-19T14:22:43.322 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:22:43.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-08-19T14:22:43.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:22:43.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-08-19T14:22:43.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:22:43.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:22:43.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-08-19T14:22:43.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:22:43.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-08-19T14:22:43.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-08-19T14:22:43.621 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509449 2024-08-19T14:22:43.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:22:43.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-08-19T14:22:43.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:22:43.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:22:43.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:22:44.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:22:44.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:22:44.217 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:22:44.217 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:22:44.217 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:22:44.217 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:22:44.217 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:22:44.217 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:22:44.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:22:44.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:22:44.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:22:44.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:22:44.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:22:44.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:22:44.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:22:44.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:454: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2024-08-19T14:22:44.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_basic: echo ABCDEF 2024-08-19T14:22:44.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-08-19T14:22:44.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-08-19T14:22:44.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-08-19T14:22:44.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-08-19T14:22:44.993 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-08-19T14:22:44.993 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:22:44.993 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:22:44.993 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:22:44.993 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:22:45.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T14:22:45.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:22:45.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T14:22:45.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T14:22:45.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:22:45.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:22:45.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T14:22:45.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:22:45.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T14:22:45.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:22:45.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:22:45.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T14:22:45.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:22:45.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T14:22:45.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:22:45.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:22:45.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:22:45.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:22:45.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:22:45.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:22:45.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T14:22:45.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:22:45.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:22:45.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T14:22:45.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:22:45.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:22:45.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-08-19T14:22:47.204 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-08-19T14:22:47.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T14:22:47.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:22:47.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:22:47.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T14:22:47.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:22:47.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:22:47.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:22:47.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:22:47.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:22:47.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:22:47.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:22:47.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:22:47.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:22:47.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:22:47.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:22:47.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:22:47.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:22:47.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:22:47.722 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:22:47.722 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:22:47.722 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:22:47.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:22:47.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:22:47.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:22:47.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:22:47.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:22:47.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:22:47.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:22:47.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:22:47.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:22:47.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:22:47.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:22:47.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:22:47.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T14:22:47.723 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T14:22:47.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:22:47.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T14:22:47.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T14:22:47.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:22:47.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:22:47.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:22:47.750 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:22:47.750+0000 7fde3ccf0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:22:47.753 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:22:47.753+0000 7fde3ccf0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:22:47.754 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:22:47.754+0000 7fde3ccf0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:22:48.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T14:22:48.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:22:48.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:22:48.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:22:48.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:22:48.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:22:48.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:22:48.002 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:22:48.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:22:48.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:22:48.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:22:48.274 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:22:48.274+0000 7fde3ccf0640 -1 Falling back to public interface 2024-08-19T14:22:49.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:22:49.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:22:49.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:22:49.268 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:22:49.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:22:49.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:22:49.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:22:50.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:22:50.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:22:50.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:22:50.540 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:22:50.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:22:50.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:22:50.642 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:22:50.641+0000 7fde3ccf0640 -1 osd.0 21 log_to_monitors true 2024-08-19T14:22:50.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:22:51.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:22:51.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:22:51.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:22:51.863 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:22:51.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:22:51.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:22:52.135 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/564487698,v1:127.0.0.1:6803/564487698] [v2:127.0.0.1:6804/564487698,v1:127.0.0.1:6805/564487698] exists,up 432c1b07-ec10-4a79-9447-2bba7c043e8e 2024-08-19T14:22:52.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:22:52.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:22:52.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:22:52.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:22:52.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:22:52.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:22:52.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:22:52.136 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:22:52.136 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:22:52.136 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:22:52.136 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:22:52.136 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:22:52.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:22:52.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:22:52.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:22:52.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:22:52.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:22:52.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:22:52.225 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:22:52.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:22:52.490 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:22:52.491 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:22:52.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:22:52.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:22:52.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:22:52.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-08-19T14:22:52.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-08-19T14:22:52.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-08-19T14:22:52.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:22:52.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:22:52.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-08-19T14:22:52.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-08-19T14:22:52.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672977' 2024-08-19T14:22:52.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:22:52.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:22:52.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-08-19T14:22:52.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-08-19T14:22:52.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672977 2-64424509452' 2024-08-19T14:22:52.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:22:52.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-08-19T14:22:52.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:22:52.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:22:52.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-08-19T14:22:52.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:22:52.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-08-19T14:22:52.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-08-19T14:22:52.800 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 107374182402 2024-08-19T14:22:52.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:22:53.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-08-19T14:22:53.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:22:54.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:22:54.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:22:54.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-08-19T14:22:54.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:22:55.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T14:22:55.344 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:22:55.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-08-19T14:22:55.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:22:55.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-08-19T14:22:55.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:22:55.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:22:55.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-08-19T14:22:55.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:22:55.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-08-19T14:22:55.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-08-19T14:22:55.621 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949672977 2024-08-19T14:22:55.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:22:55.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672977 2024-08-19T14:22:55.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:22:55.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-08-19T14:22:55.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:22:55.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:22:55.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-08-19T14:22:55.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:22:55.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-08-19T14:22:55.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-08-19T14:22:55.893 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509452 2024-08-19T14:22:55.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:22:56.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-08-19T14:22:56.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:22:56.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:22:56.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:22:56.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:22:56.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:22:56.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:22:56.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:22:56.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:22:56.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:22:56.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:22:56.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:22:56.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:22:56.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:22:56.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:22:56.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:22:57.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:22:57.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:22:57.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:22:57.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_basic: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-08-19T14:22:57.193 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: { 2024-08-19T14:22:57.193 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-08-19T14:22:57.194 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:22:57.207 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: { 2024-08-19T14:22:57.207 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-08-19T14:22:57.207 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:22:57.219 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: { 2024-08-19T14:22:57.219 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-08-19T14:22:57.219 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:22:57.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2024-08-19T14:22:57.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-08-19T14:22:57.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T14:22:57.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-08-19T14:22:57.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:22:57.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: local pgid=1.0 2024-08-19T14:22:57.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2024-08-19T14:22:57.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:22:57.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:22:57.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:22:57.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:22:57.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: local primary=1 2024-08-19T14:22:57.818 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2024-08-19T14:22:57.818 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:22:57.818 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:22:57.818 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:22:57.818 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:22:58.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2024-08-19T14:22:34.515356+0000 2024-08-19T14:22:58.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_basic: ceph tell 1.0 schedule-deep-scrub 2024-08-19T14:22:58.200 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T14:22:58.200 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": true, 2024-08-19T14:22:58.200 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T14:22:58.200 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-05T14:21:18.200457+0000" 2024-08-19T14:22:58.200 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:22:58.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2024-08-19T14:22:34.515356+0000 2024-08-19T14:22:58.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-08-19T14:22:58.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:22:34.515356+0000 2024-08-19T14:22:58.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:22:58.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:22:58.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:22:58.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:22:58.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:22:58.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:22:58.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:22:58.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:22:58.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:22:34.515356+0000 '>' 2024-08-19T14:22:34.515356+0000 2024-08-19T14:22:58.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:22:59.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:22:59.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:22:59.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:22:59.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:22:59.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:22:59.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:22:59.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:22:59.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:22:34.515356+0000 '>' 2024-08-19T14:22:34.515356+0000 2024-08-19T14:22:59.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:23:00.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:23:00.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:23:00.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:23:00.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:23:00.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:23:00.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:23:00.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:23:01.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:22:34.515356+0000 '>' 2024-08-19T14:22:34.515356+0000 2024-08-19T14:23:01.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:23:02.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:23:02.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:23:02.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:23:02.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:23:02.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:23:02.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:23:02.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:23:02.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:22:34.515356+0000 '>' 2024-08-19T14:22:34.515356+0000 2024-08-19T14:23:02.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:23:03.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:23:03.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:23:03.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:23:03.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:23:03.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:23:03.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:23:03.326 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:23:03.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:22:34.515356+0000 '>' 2024-08-19T14:22:34.515356+0000 2024-08-19T14:23:03.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:23:04.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:23:04.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:23:04.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:23:04.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:23:04.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:23:04.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:23:04.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:23:04.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:22:58.440974+0000 '>' 2024-08-19T14:22:34.515356+0000 2024-08-19T14:23:04.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:23:04.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-08-19T14:23:04.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:23:04.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:23:04.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:23:04.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:23:04.872 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:23:04.872 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:23:04.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:23:04.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:23:04.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:23:04.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:23:04.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:23:04.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:23:04.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:23:04.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:23:04.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:23:05.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:23:05.224 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:23:05.224 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:23:05.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:23:05.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:23:05.224 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:23:05.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182405 2024-08-19T14:23:05.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182405 2024-08-19T14:23:05.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405' 2024-08-19T14:23:05.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:23:05.326 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:23:05.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-08-19T14:23:05.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-08-19T14:23:05.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-42949672981' 2024-08-19T14:23:05.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:23:05.430 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:23:05.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-08-19T14:23:05.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-08-19T14:23:05.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-42949672981 2-64424509456' 2024-08-19T14:23:05.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:23:05.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182405 2024-08-19T14:23:05.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:23:05.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:23:05.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182405 2024-08-19T14:23:05.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:23:05.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182405 2024-08-19T14:23:05.534 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 107374182405 2024-08-19T14:23:05.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-08-19T14:23:05.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:23:05.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182405 2024-08-19T14:23:05.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:23:06.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:23:06.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:23:07.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182406 -lt 107374182405 2024-08-19T14:23:07.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:23:07.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-08-19T14:23:07.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:23:07.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:23:07.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-08-19T14:23:07.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:23:07.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-08-19T14:23:07.078 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672981 2024-08-19T14:23:07.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-08-19T14:23:07.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:23:07.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-08-19T14:23:07.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:23:07.344 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-08-19T14:23:07.344 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:23:07.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:23:07.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-08-19T14:23:07.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:23:07.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-08-19T14:23:07.347 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509456 2024-08-19T14:23:07.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-08-19T14:23:07.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:23:07.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-08-19T14:23:07.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:23:07.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:23:07.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:23:07.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:23:07.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:23:07.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:23:07.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:23:07.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:23:07.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:23:07.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:23:07.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:23:08.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:23:08.221 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:23:08.222 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:23:08.222 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:23:08.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:23:08.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:23:08.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:23:08.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2024-08-19T14:23:08.800 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T14:23:08.800 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-08-19T14:22:58.688171+0000 21'1 26:57 [1,0] 1 [1,0] 1 21'1 2024-08-19T14:22:58.440974+0000 21'1 2024-08-19T14:22:58.440974+0000 0 1 periodic scrub scheduled @ 2024-08-20T14:22:58.440974+0000 1 0 2024-08-19T14:23:08.800 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:23:08.800 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T14:23:08.801 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:23:08.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-08-19T14:23:08.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-08-19T14:23:08.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-08-19T14:23:08.812 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-08-19T14:23:08.812 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:23:08.812 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:23:08.812 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:23:08.812 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:23:09.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T14:23:09.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:23:09.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T14:23:09.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-08-19T14:23:09.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:23:09.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:23:09.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T14:23:09.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:23:09.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-08-19T14:23:09.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:23:09.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:23:09.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T14:23:09.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:23:09.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T14:23:09.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:23:09.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:23:09.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:23:09.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:23:09.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:23:09.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:23:09.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-08-19T14:23:09.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:23:09.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:23:09.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T14:23:09.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:23:09.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:23:09.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-08-19T14:23:10.886 INFO:tasks.workunit.client.0.smithi102.stdout:_ 2024-08-19T14:23:10.887 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 2024-08-19T14:23:11.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T14:23:11.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:23:11.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:23:11.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T14:23:11.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:23:11.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:23:11.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:23:11.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:23:11.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:23:11.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:23:11.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:23:11.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:23:11.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:23:11.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:23:11.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:23:11.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:23:11.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:23:11.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:23:11.156 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:23:11.156 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:23:11.156 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:23:11.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:23:11.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:23:11.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:23:11.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:23:11.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:23:11.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:23:11.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:23:11.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:23:11.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:23:11.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:23:11.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:23:11.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:23:11.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T14:23:11.157 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T14:23:11.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:23:11.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T14:23:11.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T14:23:11.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:23:11.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:23:11.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:23:11.185 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:11.184+0000 7fd436ce3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:23:11.186 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:11.186+0000 7fd436ce3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:23:11.188 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:11.187+0000 7fd436ce3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:23:11.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T14:23:11.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:23:11.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:23:11.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:23:11.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:23:11.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:11.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:23:11.432 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:23:11.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:11.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:11.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:11.710 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:11.710+0000 7fd436ce3640 -1 Falling back to public interface 2024-08-19T14:23:12.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:12.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:12.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:23:12.699 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:23:12.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:12.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:12.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:13.931 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:13.930+0000 7fd436ce3640 -1 osd.0 26 log_to_monitors true 2024-08-19T14:23:13.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:13.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:13.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:23:13.995 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:23:13.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:13.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:14.370 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/2699431601,v1:127.0.0.1:6803/2699431601] [v2:127.0.0.1:6804/2699431601,v1:127.0.0.1:6805/2699431601] exists,up 432c1b07-ec10-4a79-9447-2bba7c043e8e 2024-08-19T14:23:14.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:23:14.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:23:14.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:23:14.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:23:14.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:23:14.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:23:14.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:23:14.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:23:14.373 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:23:14.373 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:23:14.373 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:23:14.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:23:14.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:23:14.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:23:14.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:23:14.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:23:14.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:23:14.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:23:14.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:23:14.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:23:14.746 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:23:14.746 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:23:14.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:23:14.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:23:14.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:23:14.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-08-19T14:23:14.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-08-19T14:23:14.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-08-19T14:23:14.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:23:14.896 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:23:15.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672984 2024-08-19T14:23:15.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672984 2024-08-19T14:23:15.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672984' 2024-08-19T14:23:15.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:23:15.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:23:15.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-08-19T14:23:15.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-08-19T14:23:15.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672984 2-64424509459' 2024-08-19T14:23:15.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:23:15.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-08-19T14:23:15.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:23:15.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:23:15.227 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-08-19T14:23:15.227 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:23:15.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-08-19T14:23:15.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-08-19T14:23:15.228 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 128849018882 2024-08-19T14:23:15.228 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:23:15.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-08-19T14:23:15.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:23:16.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:23:16.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:23:16.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-08-19T14:23:16.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:23:16.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672984 2024-08-19T14:23:16.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:23:16.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:23:16.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672984 2024-08-19T14:23:16.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:23:16.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672984 2024-08-19T14:23:16.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-08-19T14:23:16.805 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949672984 2024-08-19T14:23:16.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:23:17.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672984 2024-08-19T14:23:17.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:23:17.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-08-19T14:23:17.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:23:17.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:23:17.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-08-19T14:23:17.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:23:17.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-08-19T14:23:17.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-08-19T14:23:17.084 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509459 2024-08-19T14:23:17.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:23:17.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-08-19T14:23:17.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:23:17.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:23:17.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:23:17.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:23:17.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:23:17.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:23:17.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:23:17.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:23:17.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:23:17.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:23:17.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:23:17.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-19T14:23:17.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:23:17.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:23:17.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:23:18.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-19T14:23:18.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-19T14:23:18.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:23:18.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-19T14:23:18.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:23:18.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:23:18.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:23:18.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:23:18.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:23:18.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:23:18.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:23:18.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:23:18.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:23:18.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:23:18.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:23:18.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:23:18.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:23:18.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:23:19.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:23:19.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:23:19.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:23:19.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-08-19T14:23:19.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-08-19T14:23:19.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-08-19T14:23:19.004 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-08-19T14:23:19.004 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:23:19.004 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:23:19.004 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:23:19.005 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:23:19.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T14:23:19.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:23:19.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T14:23:19.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-08-19T14:23:19.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:23:19.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:23:19.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T14:23:19.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:23:19.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-08-19T14:23:19.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:23:19.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:23:19.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T14:23:19.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:23:19.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T14:23:19.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:23:19.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:23:19.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:23:19.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:23:19.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:23:19.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:23:19.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-08-19T14:23:19.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:23:19.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:23:19.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T14:23:19.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:23:19.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:23:19.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-08-19T14:23:21.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T14:23:21.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:23:21.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:23:21.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T14:23:21.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:23:21.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:23:21.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:23:21.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:23:21.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:23:21.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:23:21.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:23:21.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:23:21.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:23:21.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:23:21.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:23:21.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:23:21.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:23:21.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:23:21.487 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:23:21.487 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:23:21.487 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:23:21.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:23:21.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:23:21.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:23:21.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:23:21.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:23:21.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:23:21.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:23:21.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:23:21.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:23:21.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:23:21.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:23:21.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:23:21.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T14:23:21.488 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T14:23:21.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:23:21.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T14:23:21.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T14:23:21.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:23:21.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:23:21.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:23:21.517 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:21.515+0000 7fc263c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:23:21.518 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:21.518+0000 7fc263c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:23:21.519 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:21.519+0000 7fc263c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:23:21.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T14:23:21.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:23:21.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:23:21.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:23:21.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:23:21.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:21.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:23:21.761 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:23:21.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:21.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:22.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:22.056 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:22.056+0000 7fc263c16640 -1 Falling back to public interface 2024-08-19T14:23:23.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:23.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:23.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:23:23.026 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:23:23.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:23.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:23.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:24.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:24.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:24.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:23:24.296 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:23:24.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:24.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:24.352 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:24.352+0000 7fc263c16640 -1 osd.0 31 log_to_monitors true 2024-08-19T14:23:24.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:25.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:25.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:25.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:23:25.604 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:23:25.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:25.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:25.877 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/860554562,v1:127.0.0.1:6803/860554562] [v2:127.0.0.1:6804/860554562,v1:127.0.0.1:6805/860554562] exists,up 432c1b07-ec10-4a79-9447-2bba7c043e8e 2024-08-19T14:23:25.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:23:25.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:23:25.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:23:25.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:23:25.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:23:25.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:23:25.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:23:25.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:23:25.878 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:23:25.878 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:23:25.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:23:25.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:23:25.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:23:25.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:23:25.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:23:25.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:23:25.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:23:25.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:23:25.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:23:26.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:23:26.229 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:23:26.229 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:23:26.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:23:26.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:23:26.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:23:26.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-08-19T14:23:26.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-08-19T14:23:26.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-08-19T14:23:26.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:23:26.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:23:26.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672987 2024-08-19T14:23:26.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672987 2024-08-19T14:23:26.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672987' 2024-08-19T14:23:26.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:23:26.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:23:26.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-08-19T14:23:26.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-08-19T14:23:26.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672987 2-64424509462' 2024-08-19T14:23:26.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:23:26.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-08-19T14:23:26.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:23:26.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:23:26.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-08-19T14:23:26.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:23:26.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-08-19T14:23:26.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-08-19T14:23:26.539 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 150323855362 2024-08-19T14:23:26.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:23:26.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-08-19T14:23:26.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:23:26.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672987 2024-08-19T14:23:26.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:23:26.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:23:26.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672987 2024-08-19T14:23:26.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:23:26.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672987 2024-08-19T14:23:26.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-08-19T14:23:26.812 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949672987 2024-08-19T14:23:26.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:23:27.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672987 -lt 42949672987 2024-08-19T14:23:27.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:23:27.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-08-19T14:23:27.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:23:27.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:23:27.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-08-19T14:23:27.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:23:27.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-08-19T14:23:27.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-08-19T14:23:27.079 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509462 2024-08-19T14:23:27.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:23:27.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-08-19T14:23:27.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:23:27.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:23:27.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:23:27.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:23:27.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:23:27.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:23:27.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:23:27.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:23:27.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:23:27.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:23:27.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:23:27.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:23:27.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:23:27.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:23:27.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:23:28.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:23:28.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:23:28.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:23:28.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T14:23:28.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-08-19T14:23:28.266 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:22:58.440+0000 7fd4d20a3640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-08-19T14:23:28.266 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:22:58.440+0000 7fd4d20a3640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-08-19T14:23:28.266 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:22:58.440+0000 7fd4d20a3640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-08-19T14:23:28.267 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:22:58.440+0000 7fd4d20a3640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-08-19T14:23:28.267 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:22:58.440+0000 7fd4d20a3640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-08-19T14:23:28.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T14:23:28.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:23:28.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:23:28.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:23:28.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:23:28.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:23:28.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:23:28.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:23:28.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:23:28.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:23:28.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:23:28.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:23:28.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:23:28.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:23:28.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:23:28.377 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:23:28.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:23:28.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:23:28.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:23:28.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:23:28.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:23:28.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:23:28.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:23:28.396 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:23:28.396 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:23:28.396 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:23:28.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:23:28.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:23:28.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:23:28.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T14:23:28.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T14:23:28.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T14:23:28.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T14:23:28.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:23:28.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:23:28.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:23:28.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:23:28.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:23:28.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:23:28.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:23:28.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:23:28.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:23:28.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:23:28.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:23:28.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:23:28.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:23:28.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:23:28.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:23:28.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:23:28.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:23:28.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:23:28.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:23:28.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:23:28.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:23:28.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:23:28.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:23:28.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:23:28.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:23:28.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:23:28.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:23:28.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:23:28.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T14:23:28.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T14:23:28.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:23:28.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:23:28.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T14:23:28.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T14:23:28.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T14:23:28.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T14:23:28.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T14:23:28.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2024-08-19T14:23:28.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2024-08-19T14:23:28.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:540: TEST_auto_repair_bluestore_failed: local poolname=testpool 2024-08-19T14:23:28.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2024-08-19T14:23:28.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T14:23:28.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T14:23:28.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T14:23:28.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T14:23:28.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T14:23:28.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-08-19T14:23:28.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T14:23:28.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:23:28.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:23:28.792 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:23:28.792 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:23:28.792 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:23:28.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:23:28.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T14:23:28.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T14:23:28.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T14:23:28.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:23:28.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:23:28.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T14:23:28.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T14:23:28.967 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:23:28.967 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:23:28.967 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:23:28.968 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:23:28.968 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:23:28.968 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:23:28.968 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:23:28.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:23:28.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T14:23:29.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T14:23:29.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:23:29.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:23:29.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T14:23:29.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T14:23:29.049 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:23:29.049 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:23:29.049 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:23:29.049 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:23:29.049 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:23:29.049 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:23:29.049 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:23:29.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:23:29.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T14:23:29.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2024-08-19T14:23:29.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T14:23:29.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T14:23:29.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T14:23:29.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T14:23:29.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T14:23:29.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T14:23:29.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T14:23:29.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:23:29.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:23:29.918 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:23:29.918 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:23:29.918 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:23:29.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:23:29.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:23:29.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:23:29.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:23:29.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: seq 0 2 2024-08-19T14:23:29.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-08-19T14:23:29.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:23:29.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:23:29.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:23:29.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T14:23:29.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:23:29.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:23:29.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:23:29.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:23:29.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:23:29.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:23:29.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:23:29.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:23:29.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:23:29.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:23:29.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:23:29.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:23:29.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:23:29.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:23:29.953 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:23:29.953 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:23:29.953 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:23:29.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:23:29.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:23:29.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:23:29.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:23:29.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:23:29.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:23:29.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:23:29.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:23:29.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:23:29.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:23:29.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:23:29.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:23:29.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:23:29.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:23:29.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:23:29.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=51d1cc78-1f89-4a12-b052-4ea6ba578d42 2024-08-19T14:23:29.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 51d1cc78-1f89-4a12-b052-4ea6ba578d42' 2024-08-19T14:23:29.956 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 51d1cc78-1f89-4a12-b052-4ea6ba578d42 2024-08-19T14:23:29.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:23:29.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhVcNmvqrZORAAEma7TBtXM3X1UPSgVcqiZg== 2024-08-19T14:23:29.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhVcNmvqrZORAAEma7TBtXM3X1UPSgVcqiZg=="}' 2024-08-19T14:23:29.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 51d1cc78-1f89-4a12-b052-4ea6ba578d42 -i td/osd-scrub-repair/0/new.json 2024-08-19T14:23:30.157 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:23:30.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T14:23:30.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBhVcNmvqrZORAAEma7TBtXM3X1UPSgVcqiZg== --osd-uuid 51d1cc78-1f89-4a12-b052-4ea6ba578d42 2024-08-19T14:23:30.200 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:30.199+0000 7f2d9d957640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:23:30.208 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:30.208+0000 7f2d9d957640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:23:30.209 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:30.209+0000 7f2d9d957640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:23:30.209 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:30.209+0000 7f2d9d957640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:23:30.210 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:30.209+0000 7f2d9d957640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:23:30.210 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:30.209+0000 7f2d9d957640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:23:30.300 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:30.299+0000 7f2d9d957640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T14:23:35.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T14:23:35.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:23:35.738 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T14:23:35.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T14:23:35.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:23:36.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T14:23:36.096 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T14:23:36.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:23:36.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:23:36.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:23:36.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:23:36.124 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:36.124+0000 7fbce7e12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:23:36.141 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:36.140+0000 7fbce7e12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:23:36.141 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:36.141+0000 7fbce7e12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:23:36.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T14:23:36.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:23:36.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:23:36.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:23:36.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:23:36.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:36.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:23:36.397 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:23:36.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:36.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:36.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:36.674 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:36.674+0000 7fbce7e12640 -1 Falling back to public interface 2024-08-19T14:23:37.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:37.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:37.663 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:23:37.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:23:37.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:37.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:37.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:38.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:38.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:38.935 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:23:38.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:23:38.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:38.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:38.974 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:23:38.973+0000 7fbce7e12640 -1 osd.0 0 log_to_monitors true 2024-08-19T14:23:39.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:40.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:40.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:40.234 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:23:40.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:23:40.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:40.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:40.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:41.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:41.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:41.513 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:23:41.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:23:41.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:41.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:41.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:42.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:42.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:42.885 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:23:42.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:23:42.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:42.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:43.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:44.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:44.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:44.161 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:23:44.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:23:44.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:44.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:44.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:45.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:45.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:45.435 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:23:45.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:23:45.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:45.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:45.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:46.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:46.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:46.737 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:23:46.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:23:46.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:46.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:47.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:48.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:48.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:48.016 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:23:48.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:23:48.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:48.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:48.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:49.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:49.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:49.299 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:23:49.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:23:49.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:49.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:49.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:50.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:50.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:50.735 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:23:50.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:23:50.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:50.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:51.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:52.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:52.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:52.171 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:23:52.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:23:52.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:52.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:52.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:53.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:53.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:53.649 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:23:53.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:23:53.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:53.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:54.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:55.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:55.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:55.008 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:23:55.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:23:55.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:55.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:55.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:56.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:56.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:56.556 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:23:56.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:23:56.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:56.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:57.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:58.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:58.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:58.081 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:23:58.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:23:58.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:58.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:23:58.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:23:59.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:23:59.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:23:59.728 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:23:59.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:23:59.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:23:59.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:24:00.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:01.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:01.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:01.028 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:24:01.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:24:01.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:01.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:24:01.308 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/893939277,v1:127.0.0.1:6803/893939277] [v2:127.0.0.1:6804/893939277,v1:127.0.0.1:6805/893939277] exists,up 51d1cc78-1f89-4a12-b052-4ea6ba578d42 2024-08-19T14:24:01.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:24:01.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:24:01.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:24:01.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-08-19T14:24:01.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:24:01.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:24:01.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:24:01.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T14:24:01.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:24:01.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:24:01.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:24:01.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:24:01.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:24:01.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:24:01.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:24:01.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:24:01.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:24:01.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:24:01.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:24:01.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:24:01.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:24:01.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:24:01.314 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:24:01.314 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:24:01.314 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:24:01.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:24:01.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:24:01.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:24:01.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:24:01.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:24:01.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:24:01.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:24:01.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:24:01.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:24:01.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:24:01.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:24:01.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:24:01.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:24:01.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:24:01.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:24:01.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3309728c-b6c0-4dae-b112-d105f76e8793 2024-08-19T14:24:01.316 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 3309728c-b6c0-4dae-b112-d105f76e8793 2024-08-19T14:24:01.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3309728c-b6c0-4dae-b112-d105f76e8793' 2024-08-19T14:24:01.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:24:01.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCBVcNmEBquExAAQwDsdrv+MOU/5pAsVj3M+g== 2024-08-19T14:24:01.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCBVcNmEBquExAAQwDsdrv+MOU/5pAsVj3M+g=="}' 2024-08-19T14:24:01.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3309728c-b6c0-4dae-b112-d105f76e8793 -i td/osd-scrub-repair/1/new.json 2024-08-19T14:24:01.677 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:24:01.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T14:24:01.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCBVcNmEBquExAAQwDsdrv+MOU/5pAsVj3M+g== --osd-uuid 3309728c-b6c0-4dae-b112-d105f76e8793 2024-08-19T14:24:01.720 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:01.719+0000 7fce01fa7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:24:01.754 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:01.753+0000 7fce01fa7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:24:01.754 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:01.754+0000 7fce01fa7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:24:01.755 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:01.755+0000 7fce01fa7640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:24:01.755 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:01.755+0000 7fce01fa7640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:24:01.755 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:01.755+0000 7fce01fa7640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:24:01.828 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:01.828+0000 7fce01fa7640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T14:24:07.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T14:24:07.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:24:07.520 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T14:24:07.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T14:24:07.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:24:07.866 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T14:24:07.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T14:24:07.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:24:07.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:24:07.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:24:07.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:24:07.895 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:07.894+0000 7f917e643640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:24:07.897 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:07.896+0000 7f917e643640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:24:07.898 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:07.898+0000 7f917e643640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:24:08.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T14:24:08.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:24:08.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:24:08.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:24:08.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:24:08.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:08.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:24:08.175 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:24:08.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:08.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:24:08.422 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:08.422+0000 7f917e643640 -1 Falling back to public interface 2024-08-19T14:24:08.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:09.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:09.440 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:24:09.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:09.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:24:09.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:09.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:24:09.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:10.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:10.715 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:24:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:24:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:10.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:24:10.747 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:10.747+0000 7f917e643640 -1 osd.1 0 log_to_monitors true 2024-08-19T14:24:10.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:11.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:11.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:11.984 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:24:11.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:24:11.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:11.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:24:12.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:13.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:13.322 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:24:13.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:13.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:24:13.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:13.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:24:13.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:14.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:14.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:14.727 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:24:14.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:24:14.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:14.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:24:15.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:16.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:16.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:16.014 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:24:16.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:24:16.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:16.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:24:16.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:17.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:17.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:17.284 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:24:17.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:24:17.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:17.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:24:17.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:18.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:18.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:18.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:24:18.597 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:24:18.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:18.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:24:18.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:19.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:19.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:19.911 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:24:19.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:24:19.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:19.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:24:20.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:21.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:21.266 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:24:21.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:21.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:24:21.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:21.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:24:21.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:22.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:22.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:22.599 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:24:22.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:24:22.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:22.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:24:23.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:24.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:24.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:24.199 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:24:24.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:24:24.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:24.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:24:24.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:25.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:25.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:25.738 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:24:25.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:24:25.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:25.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:24:26.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:27.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:27.015 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:24:27.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:27.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:24:27.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:27.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:24:27.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:28.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:28.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:28.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:24:28.591 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:24:28.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:28.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:24:29.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:30.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:30.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:30.200 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:24:30.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:24:30.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:30.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:24:30.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:31.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:31.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:31.855 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:24:31.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:24:31.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:31.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:24:32.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:33.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:33.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:33.550 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:24:33.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:24:33.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:33.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:24:33.828 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3465679176,v1:127.0.0.1:6811/3465679176] [v2:127.0.0.1:6812/3465679176,v1:127.0.0.1:6813/3465679176] exists,up 3309728c-b6c0-4dae-b112-d105f76e8793 2024-08-19T14:24:33.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:24:33.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:24:33.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:24:33.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-08-19T14:24:33.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:24:33.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:24:33.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:24:33.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T14:24:33.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:24:33.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T14:24:33.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:24:33.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:24:33.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:24:33.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:24:33.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T14:24:33.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T14:24:33.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:24:33.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:24:33.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:24:33.834 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:24:33.834 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:24:33.834 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:24:33.834 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:24:33.834 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:24:33.834 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:24:33.834 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:24:33.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:24:33.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:24:33.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:24:33.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:24:33.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:24:33.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:24:33.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:24:33.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:24:33.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:24:33.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:24:33.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:24:33.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:24:33.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T14:24:33.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:24:33.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5c0e9b81-3ea5-446a-a3c1-59b941c8e8b1 2024-08-19T14:24:33.836 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 5c0e9b81-3ea5-446a-a3c1-59b941c8e8b1 2024-08-19T14:24:33.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5c0e9b81-3ea5-446a-a3c1-59b941c8e8b1' 2024-08-19T14:24:33.836 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:24:33.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQChVcNmww+wMhAA8ju0cyCAneEUgYFziqbbXw== 2024-08-19T14:24:33.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQChVcNmww+wMhAA8ju0cyCAneEUgYFziqbbXw=="}' 2024-08-19T14:24:33.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5c0e9b81-3ea5-446a-a3c1-59b941c8e8b1 -i td/osd-scrub-repair/2/new.json 2024-08-19T14:24:34.218 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:24:34.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-08-19T14:24:34.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQChVcNmww+wMhAA8ju0cyCAneEUgYFziqbbXw== --osd-uuid 5c0e9b81-3ea5-446a-a3c1-59b941c8e8b1 2024-08-19T14:24:34.261 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:34.260+0000 7fecdc62f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:24:34.285 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:34.285+0000 7fecdc62f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:24:34.286 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:34.286+0000 7fecdc62f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:24:34.287 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:34.286+0000 7fecdc62f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:24:34.287 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:34.286+0000 7fecdc62f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:24:34.287 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:34.286+0000 7fecdc62f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:24:34.352 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:34.352+0000 7fecdc62f640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-08-19T14:24:39.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-08-19T14:24:39.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:24:39.525 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T14:24:39.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T14:24:39.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:24:39.874 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T14:24:39.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T14:24:39.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:24:39.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:24:39.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:24:39.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:24:39.903 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:39.902+0000 7f67e4c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:24:39.905 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:39.904+0000 7f67e4c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:24:39.907 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:39.905+0000 7f67e4c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:24:40.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T14:24:40.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:24:40.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T14:24:40.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:24:40.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:24:40.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:40.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:24:40.143 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:24:40.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:40.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:24:40.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:40.434 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:40.433+0000 7f67e4c14640 -1 Falling back to public interface 2024-08-19T14:24:41.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:41.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:41.410 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:24:41.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:24:41.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:41.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:24:41.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:42.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:42.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:42.686 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:24:42.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:24:42.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:42.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:24:42.776 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:24:42.775+0000 7f67e4c14640 -1 osd.2 0 log_to_monitors true 2024-08-19T14:24:42.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:43.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:43.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:43.978 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:24:43.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:24:43.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:43.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:24:44.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:45.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:45.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:45.329 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:24:45.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:24:45.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:45.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:24:45.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:46.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:46.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:46.747 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:24:46.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:24:46.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:46.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:24:47.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:48.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:48.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:48.022 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:24:48.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:24:48.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:48.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:24:48.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:49.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:49.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:49.299 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:24:49.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:24:49.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:49.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:24:49.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:50.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:50.577 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:24:50.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:50.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:24:50.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:50.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:24:50.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:51.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:51.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:51.887 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:24:51.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:24:51.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:51.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:24:52.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:53.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:53.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:53.166 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:24:53.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:24:53.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:53.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:24:53.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:54.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:54.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:54.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:24:54.441 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:24:54.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:54.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:24:55.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:56.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:56.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:56.237 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:24:56.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:24:56.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:56.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:24:56.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:57.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:57.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:57.801 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:24:57.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:24:57.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:57.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:24:58.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:24:59.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:24:59.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:24:59.561 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:24:59.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:24:59.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:24:59.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:25:00.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:25:01.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:25:01.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:25:01.151 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:25:01.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:25:01.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:25:01.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:25:01.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:25:02.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:25:02.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:25:02.751 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:25:02.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:25:02.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:25:02.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:25:03.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:25:04.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:25:04.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:25:04.076 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:25:04.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:25:04.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:25:04.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:25:04.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:25:05.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:25:05.780 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:25:05.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:25:05.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:25:05.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:25:05.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:25:06.108 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3452467206,v1:127.0.0.1:6819/3452467206] [v2:127.0.0.1:6820/3452467206,v1:127.0.0.1:6821/3452467206] exists,up 5c0e9b81-3ea5-446a-a3c1-59b941c8e8b1 2024-08-19T14:25:06.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:25:06.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:25:06.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:25:06.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2024-08-19T14:25:06.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-08-19T14:25:06.532 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'testpool' created 2024-08-19T14:25:06.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T14:25:07.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2024-08-19T14:25:07.991 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 2 2024-08-19T14:25:08.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:554: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-08-19T14:25:08.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:25:08.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:25:08.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:25:08.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:25:08.004 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:25:08.004 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:25:08.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:25:08.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:25:08.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:25:08.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:25:08.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:25:08.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:25:08.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:25:08.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:25:08.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:25:08.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:25:08.419 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:25:08.419 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:25:08.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:25:08.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:25:08.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:25:08.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836496 2024-08-19T14:25:08.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836496 2024-08-19T14:25:08.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496' 2024-08-19T14:25:08.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:25:08.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:25:08.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-08-19T14:25:08.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-08-19T14:25:08.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-42949672973' 2024-08-19T14:25:08.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:25:08.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:25:08.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-08-19T14:25:08.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-08-19T14:25:08.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-42949672973 2-64424509447' 2024-08-19T14:25:08.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:25:08.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836496 2024-08-19T14:25:08.738 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:25:08.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:25:08.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836496 2024-08-19T14:25:08.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:25:08.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836496 2024-08-19T14:25:08.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-08-19T14:25:08.740 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836496 2024-08-19T14:25:08.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:25:09.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836496 2024-08-19T14:25:09.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:25:10.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:25:10.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:25:10.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836496 2024-08-19T14:25:10.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:25:10.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-08-19T14:25:10.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:25:10.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:25:10.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-08-19T14:25:10.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:25:10.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-08-19T14:25:10.330 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672973 2024-08-19T14:25:10.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-08-19T14:25:10.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:25:10.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672973 2024-08-19T14:25:10.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:25:10.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-08-19T14:25:10.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:25:10.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:25:10.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-08-19T14:25:10.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:25:10.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-08-19T14:25:10.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-08-19T14:25:10.600 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509447 2024-08-19T14:25:10.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:25:10.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-08-19T14:25:10.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:25:10.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:25:10.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:25:11.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:25:11.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:25:11.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:25:11.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:25:11.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:25:11.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:25:11.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:25:11.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:25:11.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:25:11.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:25:11.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:25:11.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:25:11.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:25:11.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:25:11.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:25:11.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2024-08-19T14:25:11.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_failed: echo ABCDEF 2024-08-19T14:25:11.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: seq 1 10 2024-08-19T14:25:11.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-08-19T14:25:11.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:25:11.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-08-19T14:25:11.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:25:12.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-08-19T14:25:12.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:25:12.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-08-19T14:25:12.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:25:12.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-08-19T14:25:12.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:25:12.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-08-19T14:25:12.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:25:12.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-08-19T14:25:12.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:25:12.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-08-19T14:25:12.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:25:12.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-08-19T14:25:12.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:25:13.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-08-19T14:25:13.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:25:13.202 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-08-19T14:25:13.202 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-08-19T14:25:13.202 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-08-19T14:25:13.203 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-08-19T14:25:13.203 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:25:13.203 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:25:13.203 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:25:13.203 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:25:13.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T14:25:13.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:25:13.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T14:25:13.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-08-19T14:25:13.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:25:13.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:25:13.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T14:25:13.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:25:13.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-08-19T14:25:13.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:25:13.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:25:13.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T14:25:13.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:25:13.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T14:25:13.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:25:13.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:25:13.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:25:13.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:25:13.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:25:14.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:25:14.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-08-19T14:25:14.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:25:14.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:25:14.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T14:25:14.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:25:14.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:25:14.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-08-19T14:25:15.304 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:ff7b1f36:::obj1:head# 2024-08-19T14:25:15.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:25:15.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:25:15.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:25:15.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T14:25:15.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:25:15.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:25:15.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:25:15.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:25:15.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:25:15.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:25:15.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:25:15.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:25:15.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:25:15.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:25:15.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:25:15.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:25:15.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:25:15.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:25:15.823 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:25:15.823 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:25:15.824 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:25:15.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:25:15.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:25:15.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:25:15.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:25:15.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:25:15.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:25:15.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:25:15.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:25:15.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:25:15.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:25:15.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:25:15.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:25:15.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T14:25:15.825 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T14:25:15.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:25:15.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T14:25:15.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T14:25:15.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:25:15.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:25:15.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:25:15.854 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:25:15.854+0000 7fe8cf7be640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:25:15.857 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:25:15.856+0000 7fe8cf7be640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:25:15.858 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:25:15.857+0000 7fe8cf7be640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:25:16.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T14:25:16.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:25:16.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:25:16.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:25:16.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:25:16.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:25:16.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:25:16.133 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:25:16.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:25:16.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:25:16.376 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:25:16.376+0000 7fe8cf7be640 -1 Falling back to public interface 2024-08-19T14:25:16.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:25:17.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:25:17.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:25:17.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:25:17.398 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:25:17.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:25:17.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:25:17.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:25:18.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:25:18.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:25:18.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:25:18.676 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:25:18.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:25:18.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:25:18.741 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:25:18.740+0000 7fe8cf7be640 -1 osd.0 21 log_to_monitors true 2024-08-19T14:25:18.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:25:19.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:25:19.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:25:19.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:25:19.952 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:25:19.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:25:19.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:25:20.235 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/38302380,v1:127.0.0.1:6803/38302380] [v2:127.0.0.1:6804/38302380,v1:127.0.0.1:6805/38302380] exists,up 51d1cc78-1f89-4a12-b052-4ea6ba578d42 2024-08-19T14:25:20.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:25:20.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:25:20.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:25:20.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:25:20.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:25:20.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:25:20.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:25:20.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:25:20.236 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:25:20.237 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:25:20.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:25:20.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:25:20.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:25:20.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:25:20.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:25:20.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:25:20.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:25:20.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:25:20.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:25:20.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:25:20.630 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:25:20.630 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:25:20.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:25:20.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:25:20.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:25:20.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-08-19T14:25:20.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-08-19T14:25:20.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-08-19T14:25:20.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:25:20.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:25:20.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-08-19T14:25:20.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-08-19T14:25:20.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672977' 2024-08-19T14:25:20.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:25:20.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:25:20.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-08-19T14:25:20.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-08-19T14:25:20.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672977 2-64424509451' 2024-08-19T14:25:20.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:25:20.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-08-19T14:25:20.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:25:20.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:25:20.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-08-19T14:25:20.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:25:20.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-08-19T14:25:20.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-08-19T14:25:20.943 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 107374182402 2024-08-19T14:25:20.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:25:21.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-08-19T14:25:21.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:25:22.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:25:22.208 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:25:22.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-08-19T14:25:22.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:25:22.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-08-19T14:25:22.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:25:22.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:25:22.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-08-19T14:25:22.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:25:22.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-08-19T14:25:22.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-08-19T14:25:22.484 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949672977 2024-08-19T14:25:22.484 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:25:22.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-08-19T14:25:22.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:25:22.752 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-08-19T14:25:22.752 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:25:22.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:25:22.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-08-19T14:25:22.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:25:22.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-08-19T14:25:22.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-08-19T14:25:22.755 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509451 2024-08-19T14:25:22.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:25:23.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-08-19T14:25:23.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:25:23.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:25:23.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:25:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:25:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:25:23.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:25:23.354 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:25:23.354 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:25:23.354 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:25:23.354 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:25:23.354 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:25:23.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:25:23.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:25:23.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:25:23.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:25:23.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:25:23.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:25:23.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:25:23.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-08-19T14:25:23.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-08-19T14:25:23.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-08-19T14:25:23.949 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-08-19T14:25:23.949 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:25:23.949 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:25:23.949 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:25:23.949 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:25:24.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T14:25:24.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:25:24.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T14:25:24.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-08-19T14:25:24.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:25:24.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:25:24.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T14:25:24.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:25:24.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-08-19T14:25:24.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:25:24.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:25:24.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T14:25:24.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:25:24.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T14:25:24.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:25:24.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:25:24.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:25:24.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:25:24.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:25:24.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:25:24.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-08-19T14:25:24.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:25:24.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:25:24.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T14:25:24.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:25:24.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:25:24.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-08-19T14:25:25.755 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:104778fc:::obj2:head# 2024-08-19T14:25:26.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:25:26.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:25:26.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:25:26.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T14:25:26.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:25:26.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:25:26.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:25:26.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:25:26.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:25:26.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:25:26.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:25:26.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:25:26.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:25:26.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:25:26.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:25:26.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:25:26.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:25:26.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:25:26.277 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:25:26.277 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:25:26.277 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:25:26.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:25:26.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:25:26.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:25:26.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:25:26.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:25:26.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:25:26.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:25:26.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:25:26.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:25:26.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:25:26.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:25:26.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:25:26.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T14:25:26.278 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T14:25:26.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:25:26.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T14:25:26.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T14:25:26.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:25:26.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:25:26.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:25:26.310 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:25:26.309+0000 7f0fa5212640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:25:26.312 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:25:26.312+0000 7f0fa5212640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:25:26.314 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:25:26.313+0000 7f0fa5212640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:25:26.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T14:25:26.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:25:26.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:25:26.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:25:26.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:25:26.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:25:26.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:25:26.559 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:25:26.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:25:26.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:25:26.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:25:26.836 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:25:26.836+0000 7f0fa5212640 -1 Falling back to public interface 2024-08-19T14:25:27.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:25:27.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:25:27.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:25:27.825 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:25:27.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:25:27.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:25:28.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:25:29.087 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:25:29.085+0000 7f0fa5212640 -1 osd.0 26 log_to_monitors true 2024-08-19T14:25:29.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:25:29.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:25:29.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:25:29.131 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:25:29.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:25:29.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:25:29.518 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/3759772032,v1:127.0.0.1:6803/3759772032] [v2:127.0.0.1:6804/3759772032,v1:127.0.0.1:6805/3759772032] exists,up 51d1cc78-1f89-4a12-b052-4ea6ba578d42 2024-08-19T14:25:29.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:25:29.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:25:29.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:25:29.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:25:29.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:25:29.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:25:29.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:25:29.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:25:29.519 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:25:29.519 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:25:29.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:25:29.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:25:29.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:25:29.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:25:29.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:25:29.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:25:29.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:25:29.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:25:29.607 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:25:29.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:25:29.875 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:25:29.875 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:25:29.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:25:29.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:25:29.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:25:30.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-08-19T14:25:30.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-08-19T14:25:30.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-08-19T14:25:30.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:25:30.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:25:30.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-08-19T14:25:30.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-08-19T14:25:30.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672980' 2024-08-19T14:25:30.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:25:30.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:25:30.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-08-19T14:25:30.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-08-19T14:25:30.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672980 2-64424509453' 2024-08-19T14:25:30.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:25:30.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-08-19T14:25:30.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:25:30.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:25:30.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-08-19T14:25:30.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:25:30.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-08-19T14:25:30.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-08-19T14:25:30.232 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 128849018882 2024-08-19T14:25:30.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:25:30.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-08-19T14:25:30.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:25:31.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:25:31.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:25:31.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-08-19T14:25:31.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:25:32.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T14:25:32.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:25:33.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-08-19T14:25:33.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:25:33.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-08-19T14:25:33.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:25:33.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:25:33.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-08-19T14:25:33.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:25:33.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-08-19T14:25:33.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-08-19T14:25:33.058 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949672980 2024-08-19T14:25:33.058 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:25:33.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-08-19T14:25:33.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:25:33.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-08-19T14:25:33.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:25:33.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:25:33.326 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-08-19T14:25:33.326 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:25:33.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-08-19T14:25:33.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-08-19T14:25:33.327 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509453 2024-08-19T14:25:33.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:25:33.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509453 2024-08-19T14:25:33.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:25:33.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:25:33.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:25:33.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:25:33.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:25:33.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:25:33.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:25:33.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:25:33.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:25:33.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:25:33.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:25:34.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-19T14:25:34.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:25:34.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:25:34.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:25:34.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-19T14:25:34.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-19T14:25:34.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:25:34.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-19T14:25:34.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:25:34.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:25:34.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:25:34.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:25:34.617 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:25:34.617 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:25:34.617 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:25:34.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:25:34.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:25:34.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:25:34.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-19T14:25:34.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:25:34.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:25:34.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:25:35.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-19T14:25:35.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-19T14:25:35.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T14:25:35.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T14:25:35.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T14:25:35.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T14:25:35.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T14:25:35.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T14:25:35.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T14:25:35.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T14:25:35.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T14:25:35.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-19T14:25:35.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:25:35.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-19T14:25:35.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:25:35.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:25:35.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:25:35.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:25:35.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:25:35.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:25:35.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:25:35.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:25:36.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:25:36.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:25:36.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:25:36.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:25:36.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:25:36.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:25:36.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:25:36.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-08-19T14:25:36.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:25:36.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:25:36.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:25:36.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:25:36.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-08-19T14:25:36.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:25:36.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:25:36.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T14:25:36.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:25:36.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-08-19T14:25:36.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:25:36.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:25:36.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T14:25:36.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:25:36.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T14:25:36.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:25:36.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:25:36.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:25:36.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:25:36.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:25:36.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:25:36.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-08-19T14:25:36.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:25:36.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:25:36.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T14:25:36.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:25:36.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:25:36.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-08-19T14:25:38.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:25:38.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:25:38.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:25:38.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:25:38.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:25:38.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:25:38.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:25:38.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:25:38.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:25:38.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:25:38.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:25:38.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:25:38.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:25:38.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:25:38.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:25:38.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:25:38.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:25:38.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:25:38.797 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:25:38.797 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:25:38.797 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:25:38.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:25:38.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:25:38.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:25:38.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:25:38.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:25:38.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:25:38.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:25:38.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:25:38.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:25:38.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:25:38.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:25:38.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:25:38.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:25:38.798 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T14:25:38.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:25:38.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T14:25:38.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:25:38.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:25:38.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:25:38.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:25:38.828 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:25:38.827+0000 7f8e02418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:25:38.830 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:25:38.830+0000 7f8e02418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:25:38.831 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:25:38.831+0000 7f8e02418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:25:39.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:25:39.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:25:39.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:25:39.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:25:39.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:25:39.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:25:39.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:25:39.072 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:25:39.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:25:39.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:25:39.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:25:39.354 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:25:39.354+0000 7f8e02418640 -1 Falling back to public interface 2024-08-19T14:25:40.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:25:40.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:25:40.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:25:40.340 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:25:40.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:25:40.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:25:40.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:25:41.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:25:41.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:25:41.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:25:41.615 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:25:41.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:25:41.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:25:41.704 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:25:41.704+0000 7f8e02418640 -1 osd.1 31 log_to_monitors true 2024-08-19T14:25:41.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:25:42.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:25:42.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:25:42.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:25:42.893 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:25:42.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:25:42.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:25:43.161 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/1459376740,v1:127.0.0.1:6811/1459376740] [v2:127.0.0.1:6812/1459376740,v1:127.0.0.1:6813/1459376740] exists,up 3309728c-b6c0-4dae-b112-d105f76e8793 2024-08-19T14:25:43.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:25:43.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:25:43.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:25:43.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:25:43.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:25:43.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:25:43.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:25:43.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:25:43.162 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:25:43.162 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:25:43.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:25:43.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:25:43.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:25:43.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:25:43.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:25:43.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:25:43.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:25:43.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:25:43.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:25:43.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:25:43.513 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:25:43.513 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:25:43.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:25:43.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:25:43.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:25:43.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018885 2024-08-19T14:25:43.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018885 2024-08-19T14:25:43.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885' 2024-08-19T14:25:43.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:25:43.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:25:43.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-08-19T14:25:43.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-08-19T14:25:43.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-150323855362' 2024-08-19T14:25:43.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:25:43.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:25:43.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-08-19T14:25:43.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-08-19T14:25:43.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-150323855362 2-64424509457' 2024-08-19T14:25:43.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:25:43.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018885 2024-08-19T14:25:43.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:25:43.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:25:43.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018885 2024-08-19T14:25:43.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:25:43.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018885 2024-08-19T14:25:43.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2024-08-19T14:25:43.873 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 128849018885 2024-08-19T14:25:43.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:25:44.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018885 -lt 128849018885 2024-08-19T14:25:44.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:25:44.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855362 2024-08-19T14:25:44.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:25:44.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:25:44.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855362 2024-08-19T14:25:44.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:25:44.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-08-19T14:25:44.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-08-19T14:25:44.105 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 150323855362 2024-08-19T14:25:44.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:25:44.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-08-19T14:25:44.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:25:44.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-08-19T14:25:44.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:25:44.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:25:44.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-08-19T14:25:44.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:25:44.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-08-19T14:25:44.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-08-19T14:25:44.407 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509457 2024-08-19T14:25:44.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:25:44.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-08-19T14:25:44.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:25:44.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:25:44.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:25:45.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:25:45.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:25:45.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:25:45.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:25:45.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:25:45.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:25:45.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:25:45.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:25:45.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:25:45.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:25:45.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:25:45.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:25:45.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:25:45.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:25:45.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:25:45.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2024-08-19T14:25:45.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-08-19T14:25:45.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-08-19T14:25:45.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-08-19T14:25:45.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:25:45.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: local pgid=1.0 2024-08-19T14:25:45.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2024-08-19T14:25:45.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:25:45.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-08-19T14:25:45.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-08-19T14:25:45.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:25:46.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: local primary=1 2024-08-19T14:25:46.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2024-08-19T14:25:46.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:25:46.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:25:46.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:25:46.182 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:25:46.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2024-08-19T14:25:06.415166+0000 2024-08-19T14:25:46.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:574: TEST_auto_repair_bluestore_failed: ceph tell 1.0 schedule-deep-scrub 2024-08-19T14:25:46.536 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T14:25:46.537 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": true, 2024-08-19T14:25:46.537 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T14:25:46.537 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-05T14:24:06.537189+0000" 2024-08-19T14:25:46.537 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:25:46.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:577: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2024-08-19T14:25:06.415166+0000 2024-08-19T14:25:46.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-08-19T14:25:46.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:25:06.415166+0000 2024-08-19T14:25:46.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:25:46.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:25:46.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:25:46.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:25:46.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:25:46.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:25:46.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:25:46.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:25:46.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:25:06.415166+0000 '>' 2024-08-19T14:25:06.415166+0000 2024-08-19T14:25:46.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:25:47.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:25:47.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:25:47.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:25:47.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:25:47.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:25:47.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:25:47.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:25:48.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T14:24:06.537189+0000 '>' 2024-08-19T14:25:06.415166+0000 2024-08-19T14:25:48.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:25:49.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:25:49.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:25:49.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:25:49.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:25:49.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:25:49.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:25:49.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:25:49.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T14:24:06.537189+0000 '>' 2024-08-19T14:25:06.415166+0000 2024-08-19T14:25:49.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:25:50.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:25:50.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:25:50.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:25:50.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:25:50.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:25:50.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:25:50.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:25:50.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T14:24:06.537189+0000 '>' 2024-08-19T14:25:06.415166+0000 2024-08-19T14:25:50.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:25:51.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:25:51.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:25:51.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:25:51.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:25:51.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:25:51.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:25:51.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:25:51.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T14:24:06.537189+0000 '>' 2024-08-19T14:25:06.415166+0000 2024-08-19T14:25:51.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:25:52.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:25:52.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:25:52.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:25:52.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:25:52.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:25:52.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:25:52.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:25:53.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T14:24:06.537189+0000 '>' 2024-08-19T14:25:06.415166+0000 2024-08-19T14:25:53.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:25:54.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:25:54.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:25:54.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:25:54.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:25:54.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:25:54.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:25:54.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:25:54.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:25:51.041098+0000 '>' 2024-08-19T14:25:06.415166+0000 2024-08-19T14:25:54.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:25:54.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-08-19T14:25:54.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:25:54.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:25:54.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:25:54.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:25:54.469 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:25:54.469 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:25:54.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:25:54.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:25:54.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:25:54.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:25:54.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:25:54.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:25:54.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:25:54.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:25:54.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:25:54.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:25:54.822 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:25:54.823 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:25:54.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:25:54.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:25:54.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:25:54.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018889 2024-08-19T14:25:54.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018889 2024-08-19T14:25:54.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018889' 2024-08-19T14:25:54.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:25:54.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:25:55.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-08-19T14:25:55.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-08-19T14:25:55.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018889 1-150323855365' 2024-08-19T14:25:55.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:25:55.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:25:55.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-08-19T14:25:55.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-08-19T14:25:55.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018889 1-150323855365 2-64424509460' 2024-08-19T14:25:55.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:25:55.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018889 2024-08-19T14:25:55.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:25:55.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:25:55.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018889 2024-08-19T14:25:55.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:25:55.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018889 2024-08-19T14:25:55.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018889' 2024-08-19T14:25:55.135 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 128849018889 2024-08-19T14:25:55.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:25:55.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018889 2024-08-19T14:25:55.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:25:56.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:25:56.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:25:56.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018889 -lt 128849018889 2024-08-19T14:25:56.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:25:56.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855365 2024-08-19T14:25:56.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:25:56.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:25:56.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855365 2024-08-19T14:25:56.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:25:56.676 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 150323855365 2024-08-19T14:25:56.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-08-19T14:25:56.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855365' 2024-08-19T14:25:56.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:25:56.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-08-19T14:25:56.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:25:56.979 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-08-19T14:25:56.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:25:56.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:25:56.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-08-19T14:25:56.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:25:56.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-08-19T14:25:56.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-08-19T14:25:56.982 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509460 2024-08-19T14:25:56.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:25:57.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509460 2024-08-19T14:25:57.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:25:57.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:25:57.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:25:57.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:25:57.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:25:57.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:25:57.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:25:57.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:25:57.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:25:57.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:25:57.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:25:57.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:25:57.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:25:57.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:25:57.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:25:58.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:25:58.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:25:58.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:25:58.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-08-19T14:25:58.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:25:58.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:25:58.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:25:58.463 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:25:58.463 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:25:58.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:25:58.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:25:58.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:25:58.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018890 2024-08-19T14:25:58.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018890 2024-08-19T14:25:58.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890' 2024-08-19T14:25:58.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:25:58.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:25:58.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855367 2024-08-19T14:25:58.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855367 2024-08-19T14:25:58.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890 1-150323855367' 2024-08-19T14:25:58.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:25:58.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:25:58.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-08-19T14:25:58.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-08-19T14:25:58.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890 1-150323855367 2-64424509462' 2024-08-19T14:25:58.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:25:58.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018890 2024-08-19T14:25:58.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:25:58.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:25:58.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018890 2024-08-19T14:25:58.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:25:58.771 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 128849018890 2024-08-19T14:25:58.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018890 2024-08-19T14:25:58.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018890' 2024-08-19T14:25:58.772 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:25:59.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018889 -lt 128849018890 2024-08-19T14:25:59.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:26:00.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:26:00.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:26:00.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018891 -lt 128849018890 2024-08-19T14:26:00.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:26:00.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855367 2024-08-19T14:26:00.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:26:00.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:26:00.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855367 2024-08-19T14:26:00.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:26:00.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855367 2024-08-19T14:26:00.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855367' 2024-08-19T14:26:00.348 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 150323855367 2024-08-19T14:26:00.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:26:00.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855367 -lt 150323855367 2024-08-19T14:26:00.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:26:00.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-08-19T14:26:00.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:26:00.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:26:00.616 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-08-19T14:26:00.616 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:26:00.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-08-19T14:26:00.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-08-19T14:26:00.617 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509462 2024-08-19T14:26:00.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:26:00.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-08-19T14:26:00.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-08-19T14:26:00.927 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:50.841+0000 7f8de5e0d640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-08-19T14:26:00.928 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:50.841+0000 7f8de5e0d640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-08-19T14:26:00.928 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:50.841+0000 7f8de5e0d640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-08-19T14:26:00.928 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:50.841+0000 7f8de5e0d640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-08-19T14:26:00.928 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:50.841+0000 7f8de5e0d640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-08-19T14:26:00.928 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:51.040+0000 7f8de5e0d640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-08-19T14:26:00.928 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:51.040+0000 7f8de5e0d640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-08-19T14:26:00.928 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:51.040+0000 7f8de5e0d640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-08-19T14:26:00.928 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:51.040+0000 7f8de5e0d640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-08-19T14:26:00.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2024-08-19T14:26:00.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-08-19T14:26:01.218 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T14:26:01.218 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-08-19T14:25:51.041175+0000 21'10 36:96 [1,0] 1 [1,0] 1 21'10 2024-08-19T14:25:51.041098+0000 21'10 2024-08-19T14:25:51.041098+0000 0 1 periodic scrub scheduled @ 2024-08-20T14:25:51.041098+0000 10 0 2024-08-19T14:26:01.218 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:26:01.218 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T14:26:01.219 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:26:01.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-08-19T14:26:01.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2024-08-19T14:26:01.477 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:26:01.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2024-08-19T14:26:01.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-08-19T14:26:01.490 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-08-19T14:26:01.490 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-08-19T14:26:01.490 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:26:01.490 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-08-19T14:26:01.490 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-08-19T14:26:01.490 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:26:01.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T14:26:01.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-08-19T14:26:01.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T14:26:02.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2024-08-19T14:26:02.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:26:02.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:26:02.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T14:26:02.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:26:02.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2024-08-19T14:26:02.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:26:02.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:26:02.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T14:26:02.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:26:02.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T14:26:02.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:26:02.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:26:02.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:26:02.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:26:02.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:26:02.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:26:02.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2024-08-19T14:26:02.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:26:02.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:26:02.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T14:26:02.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:26:02.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:26:02.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2024-08-19T14:26:03.522 INFO:tasks.workunit.client.0.smithi102.stdout:_ 2024-08-19T14:26:03.522 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 2024-08-19T14:26:03.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:26:03.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:26:03.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:26:03.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T14:26:03.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:26:03.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:26:03.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:26:03.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:26:03.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:26:03.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:26:03.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:26:03.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:26:03.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:26:03.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:26:03.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:26:03.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:26:03.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:26:03.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:26:03.791 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:26:03.791 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:26:03.791 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:26:03.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:26:03.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:26:03.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:26:03.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:26:03.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:26:03.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:26:03.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:26:03.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:26:03.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:26:03.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:26:03.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:26:03.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:26:03.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T14:26:03.793 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T14:26:03.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:26:03.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T14:26:03.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T14:26:03.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:26:03.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:26:03.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:26:03.823 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:03.822+0000 7ff273321640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:26:03.825 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:03.825+0000 7ff273321640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:26:03.826 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:03.826+0000 7ff273321640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:26:04.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T14:26:04.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:26:04.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:26:04.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:26:04.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:26:04.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:04.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:26:04.107 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:26:04.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:04.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:26:04.346 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:04.346+0000 7ff273321640 -1 Falling back to public interface 2024-08-19T14:26:04.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:26:05.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:26:05.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:05.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:26:05.377 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:26:05.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:05.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:26:05.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:26:06.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:26:06.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:06.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:26:06.648 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:26:06.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:06.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:26:06.675 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:06.675+0000 7ff273321640 -1 osd.0 36 log_to_monitors true 2024-08-19T14:26:06.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:26:07.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:26:07.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:07.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:26:07.933 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:26:07.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:07.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:26:08.265 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 40 up_thru 32 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6802/3435026971,v1:127.0.0.1:6803/3435026971] [v2:127.0.0.1:6804/3435026971,v1:127.0.0.1:6805/3435026971] exists,up 51d1cc78-1f89-4a12-b052-4ea6ba578d42 2024-08-19T14:26:08.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:26:08.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:26:08.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:26:08.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:26:08.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:26:08.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:26:08.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:26:08.265 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:26:08.266 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:26:08.266 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:26:08.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:26:08.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:26:08.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:26:08.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:26:08.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:26:08.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:26:08.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:26:08.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:26:08.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:26:08.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:26:08.645 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:26:08.645 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:26:08.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:26:08.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:26:08.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:26:08.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-08-19T14:26:08.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-08-19T14:26:08.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842' 2024-08-19T14:26:08.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:26:08.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:26:08.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855370 2024-08-19T14:26:08.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855370 2024-08-19T14:26:08.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855370' 2024-08-19T14:26:08.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:26:08.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:26:09.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509465 2024-08-19T14:26:09.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509465 2024-08-19T14:26:09.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855370 2-64424509465' 2024-08-19T14:26:09.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:26:09.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691842 2024-08-19T14:26:09.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:26:09.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:26:09.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691842 2024-08-19T14:26:09.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:26:09.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-08-19T14:26:09.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2024-08-19T14:26:09.005 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 171798691842 2024-08-19T14:26:09.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:26:09.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 171798691842 2024-08-19T14:26:09.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:26:10.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:26:10.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:26:10.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-08-19T14:26:10.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:26:10.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855370 2024-08-19T14:26:10.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:26:10.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:26:10.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855370 2024-08-19T14:26:10.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:26:10.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855370 2024-08-19T14:26:10.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855370' 2024-08-19T14:26:10.545 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 150323855370 2024-08-19T14:26:10.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:26:10.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855370 -lt 150323855370 2024-08-19T14:26:10.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:26:10.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509465 2024-08-19T14:26:10.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:26:10.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:26:10.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509465 2024-08-19T14:26:10.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:26:10.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509465 2024-08-19T14:26:10.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-08-19T14:26:10.816 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509465 2024-08-19T14:26:10.816 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:26:11.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509465 2024-08-19T14:26:11.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:26:11.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:26:11.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:26:11.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:26:11.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:26:11.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:26:11.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:26:11.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:26:11.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:26:11.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:26:11.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:26:11.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:26:11.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:26:11.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:26:11.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:26:12.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:26:12.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:26:12.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:26:12.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2024-08-19T14:26:12.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T14:26:12.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-08-19T14:26:12.036 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:50.841+0000 7f8de5e0d640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-08-19T14:26:12.036 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:50.841+0000 7f8de5e0d640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-08-19T14:26:12.036 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:50.841+0000 7f8de5e0d640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-08-19T14:26:12.036 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:50.841+0000 7f8de5e0d640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-08-19T14:26:12.036 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:50.841+0000 7f8de5e0d640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-08-19T14:26:12.036 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:51.040+0000 7f8de5e0d640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-08-19T14:26:12.037 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:51.040+0000 7f8de5e0d640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-08-19T14:26:12.037 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:51.040+0000 7f8de5e0d640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-08-19T14:26:12.037 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:51.040+0000 7f8de5e0d640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-08-19T14:26:12.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-08-19T14:26:12.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:26:12.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:26:12.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:26:12.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:26:12.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2024-08-19T14:26:12.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:26:12.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:26:12.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T14:26:12.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:26:12.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2024-08-19T14:26:12.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:26:12.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:26:12.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T14:26:12.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:26:12.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T14:26:12.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:26:12.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:26:12.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:26:12.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:26:12.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:26:12.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:26:12.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2024-08-19T14:26:12.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:26:12.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:26:12.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T14:26:12.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:26:12.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:26:12.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2024-08-19T14:26:13.734 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2024-08-19T14:26:13.735 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:104778fc:::obj2:head# 2024-08-19T14:26:14.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:26:14.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:26:14.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:26:14.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:26:14.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:26:14.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:26:14.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:26:14.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:26:14.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:26:14.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:26:14.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:26:14.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:26:14.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:26:14.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:26:14.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:26:14.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:26:14.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:26:14.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:26:14.254 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:26:14.254 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:26:14.254 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:26:14.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:26:14.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:26:14.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:26:14.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:26:14.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:26:14.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:26:14.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:26:14.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:26:14.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:26:14.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:26:14.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:26:14.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:26:14.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:26:14.255 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T14:26:14.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:26:14.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T14:26:14.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:26:14.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:26:14.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:26:14.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:26:14.285 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:14.285+0000 7f27f3cee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:26:14.287 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:14.287+0000 7f27f3cee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:26:14.289 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:14.289+0000 7f27f3cee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:26:14.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:26:14.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:26:14.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:26:14.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:26:14.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:26:14.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:14.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:26:14.540 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:26:14.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:14.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:26:14.803 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:14.803+0000 7f27f3cee640 -1 Falling back to public interface 2024-08-19T14:26:14.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:26:15.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:26:15.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:15.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:26:15.844 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:26:15.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:15.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:26:16.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:26:17.091 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:17.089+0000 7f27f3cee640 -1 osd.1 41 log_to_monitors true 2024-08-19T14:26:17.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:26:17.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:17.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:26:17.143 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:26:17.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:17.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:26:17.479 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 45 up_thru 40 down_at 42 last_clean_interval [35,41) [v2:127.0.0.1:6810/654690986,v1:127.0.0.1:6811/654690986] [v2:127.0.0.1:6812/654690986,v1:127.0.0.1:6813/654690986] exists,up 3309728c-b6c0-4dae-b112-d105f76e8793 2024-08-19T14:26:17.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:26:17.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:26:17.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:26:17.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:26:17.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:26:17.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:26:17.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:26:17.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:26:17.481 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:26:17.481 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:26:17.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:26:17.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:26:17.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:26:17.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:26:17.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:26:17.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:26:17.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:26:17.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:26:17.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:26:17.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:26:17.874 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:26:17.875 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:26:17.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:26:17.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:26:17.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:26:18.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-08-19T14:26:18.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-08-19T14:26:18.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845' 2024-08-19T14:26:18.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:26:18.014 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:26:18.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528322 2024-08-19T14:26:18.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528322 2024-08-19T14:26:18.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322' 2024-08-19T14:26:18.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:26:18.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:26:18.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509468 2024-08-19T14:26:18.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509468 2024-08-19T14:26:18.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322 2-64424509468' 2024-08-19T14:26:18.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:26:18.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691845 2024-08-19T14:26:18.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:26:18.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:26:18.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691845 2024-08-19T14:26:18.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:26:18.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-08-19T14:26:18.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691845' 2024-08-19T14:26:18.245 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 171798691845 2024-08-19T14:26:18.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:26:18.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691844 -lt 171798691845 2024-08-19T14:26:18.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:26:19.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:26:19.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:26:19.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691844 -lt 171798691845 2024-08-19T14:26:19.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:26:20.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T14:26:20.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:26:21.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-08-19T14:26:21.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:26:21.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528322 2024-08-19T14:26:21.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:26:21.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:26:21.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528322 2024-08-19T14:26:21.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:26:21.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528322 2024-08-19T14:26:21.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528322' 2024-08-19T14:26:21.061 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 193273528322 2024-08-19T14:26:21.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:26:21.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528322 -lt 193273528322 2024-08-19T14:26:21.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:26:21.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509468 2024-08-19T14:26:21.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:26:21.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:26:21.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509468 2024-08-19T14:26:21.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:26:21.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509468 2024-08-19T14:26:21.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-08-19T14:26:21.333 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509468 2024-08-19T14:26:21.333 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:26:21.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509468 -lt 64424509468 2024-08-19T14:26:21.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:26:21.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:26:21.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:26:21.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:26:21.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:26:21.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:26:21.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:26:21.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:26:21.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:26:21.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:26:21.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:26:22.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:26:22.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:26:22.195 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:26:22.195 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:26:22.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:26:22.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:26:22.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:26:22.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_failed: repair 1.0 2024-08-19T14:26:22.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-08-19T14:26:22.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-08-19T14:26:22.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:26:22.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:26:22.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:26:22.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:26:22.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T14:25:51.041098+0000 2024-08-19T14:26:22.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-08-19T14:26:23.044 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 1.0 on osd.1 to repair 2024-08-19T14:26:23.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-08-19T14:25:51.041098+0000 2024-08-19T14:26:23.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-08-19T14:26:23.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:25:51.041098+0000 2024-08-19T14:26:23.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:26:23.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:26:23.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:26:23.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:26:23.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:26:23.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:26:23.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:26:23.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:26:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:25:51.041098+0000 '>' 2024-08-19T14:25:51.041098+0000 2024-08-19T14:26:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:26:24.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:26:24.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:26:24.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:26:24.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:26:24.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:26:24.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:26:24.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:26:24.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:25:51.041098+0000 '>' 2024-08-19T14:25:51.041098+0000 2024-08-19T14:26:24.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:26:25.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:26:25.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:26:25.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:26:25.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:26:25.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:26:25.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:26:25.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:26:25.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:25:51.041098+0000 '>' 2024-08-19T14:25:51.041098+0000 2024-08-19T14:26:25.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:26:26.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:26:26.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:26:26.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:26:26.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:26:26.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:26:26.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:26:26.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:26:27.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:25:51.041098+0000 '>' 2024-08-19T14:25:51.041098+0000 2024-08-19T14:26:27.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:26:28.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:26:28.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:26:28.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:26:28.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:26:28.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:26:28.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:26:28.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:26:28.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:26:23.157778+0000 '>' 2024-08-19T14:25:51.041098+0000 2024-08-19T14:26:28.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:26:28.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:595: TEST_auto_repair_bluestore_failed: sleep 2 2024-08-19T14:26:30.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:597: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-08-19T14:26:30.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:26:30.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:26:30.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:26:30.690 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:26:30.690 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:26:30.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:26:30.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:26:30.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:26:30.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691848 2024-08-19T14:26:30.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691848 2024-08-19T14:26:30.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691848' 2024-08-19T14:26:30.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:26:30.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:26:30.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528325 2024-08-19T14:26:30.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528325 2024-08-19T14:26:30.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691848 1-193273528325' 2024-08-19T14:26:30.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:26:30.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:26:30.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509472 2024-08-19T14:26:30.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509472 2024-08-19T14:26:30.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691848 1-193273528325 2-64424509472' 2024-08-19T14:26:30.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:26:30.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691848 2024-08-19T14:26:30.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:26:30.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:26:30.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691848 2024-08-19T14:26:30.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:26:31.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691848 2024-08-19T14:26:31.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691848' 2024-08-19T14:26:31.000 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 171798691848 2024-08-19T14:26:31.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:26:31.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691847 -lt 171798691848 2024-08-19T14:26:31.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:26:32.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:26:32.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:26:32.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691848 -lt 171798691848 2024-08-19T14:26:32.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:26:32.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528325 2024-08-19T14:26:32.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:26:32.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:26:32.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528325 2024-08-19T14:26:32.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:26:32.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528325 2024-08-19T14:26:32.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528325' 2024-08-19T14:26:32.542 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 193273528325 2024-08-19T14:26:32.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:26:32.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528325 -lt 193273528325 2024-08-19T14:26:32.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:26:32.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509472 2024-08-19T14:26:32.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:26:32.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:26:32.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509472 2024-08-19T14:26:32.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:26:32.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509472 2024-08-19T14:26:32.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2024-08-19T14:26:32.810 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509472 2024-08-19T14:26:32.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:26:33.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509472 -lt 64424509472 2024-08-19T14:26:33.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-08-19T14:26:33.325 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T14:26:33.326 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2024-08-19T14:26:23.157868+0000 21'10 46:133 [1,0] 1 [1,0] 1 21'10 2024-08-19T14:26:23.157778+0000 21'10 2024-08-19T14:26:23.157778+0000 0 1 periodic scrub scheduled @ 2024-08-20T14:26:23.157778+0000 9 0 2024-08-19T14:26:33.326 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:26:33.326 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T14:26:33.326 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:26:33.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-08-19T14:26:33.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2024-08-19T14:26:33.585 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:26:33.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:600: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-08-19T14:26:33.599 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:50.841+0000 7f8de5e0d640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-08-19T14:26:33.599 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:50.841+0000 7f8de5e0d640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-08-19T14:26:33.599 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:50.841+0000 7f8de5e0d640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-08-19T14:26:33.599 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:50.841+0000 7f8de5e0d640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-08-19T14:26:33.599 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:50.841+0000 7f8de5e0d640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-08-19T14:26:33.599 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:51.040+0000 7f8de5e0d640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-08-19T14:26:33.599 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:51.040+0000 7f8de5e0d640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-08-19T14:26:33.599 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:51.040+0000 7f8de5e0d640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-08-19T14:26:33.599 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:25:51.040+0000 7f8de5e0d640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-08-19T14:26:33.599 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:26:23.156+0000 7f27d7c67640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] MUST_REPAIR planned AUTO_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] scrubber: scrub_finish before flags: AUTO_REPAIR REQ_SCRUB. repair state: repair. deep_scrub_on_error: 0 2024-08-19T14:26:33.600 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:26:23.156+0000 7f27d7c67640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-08-19T14:26:33.600 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:26:23.156+0000 7f27d7c67640 15 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-08-19T14:26:33.600 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:26:23.156+0000 7f27d7c67640 20 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish All may be fixed 2024-08-19T14:26:33.600 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:26:23.156+0000 7f27d7c67640 19 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-08-19T14:26:33.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T14:26:33.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:26:33.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:26:33.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:26:33.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:26:33.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:26:33.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:26:33.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:26:33.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:26:33.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:26:33.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:26:33.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:26:33.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:26:33.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:26:33.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:26:33.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:26:33.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:26:33.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:26:33.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:26:33.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:26:33.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:26:33.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:26:33.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:26:33.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:26:33.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:26:33.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:26:33.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:26:33.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:26:33.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:26:33.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T14:26:33.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T14:26:33.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T14:26:33.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T14:26:33.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:26:33.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:26:33.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:26:33.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:26:33.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:26:33.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:26:33.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:26:33.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:26:33.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:26:33.738 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:26:33.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:26:33.738 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:26:33.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:26:33.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:26:33.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:26:33.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:26:33.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:26:33.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:26:33.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:26:33.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:26:33.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:26:33.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:26:33.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:26:33.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:26:33.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:26:33.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:26:33.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:26:33.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:26:33.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T14:26:33.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T14:26:33.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:26:33.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:26:33.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T14:26:33.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T14:26:33.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T14:26:33.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T14:26:33.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T14:26:33.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2024-08-19T14:26:33.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:604: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2024-08-19T14:26:33.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:605: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2024-08-19T14:26:33.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2024-08-19T14:26:33.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T14:26:33.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T14:26:33.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T14:26:33.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T14:26:33.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T14:26:33.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-08-19T14:26:34.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T14:26:34.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:26:34.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:26:34.158 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:26:34.158 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:26:34.158 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:26:34.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:26:34.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T14:26:34.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T14:26:34.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T14:26:34.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:26:34.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:26:34.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T14:26:34.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T14:26:34.348 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:26:34.348 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:26:34.348 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:26:34.348 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:26:34.348 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:26:34.348 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:26:34.348 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:26:34.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:26:34.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T14:26:34.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T14:26:34.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:26:34.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:26:34.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T14:26:34.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T14:26:34.427 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:26:34.427 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:26:34.427 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:26:34.428 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:26:34.428 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:26:34.428 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:26:34.428 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:26:34.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:26:34.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T14:26:34.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:609: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2024-08-19T14:26:34.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T14:26:34.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T14:26:34.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T14:26:34.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T14:26:34.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T14:26:34.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T14:26:35.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T14:26:35.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:26:35.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:26:35.298 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:26:35.298 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:26:35.299 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:26:35.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:26:35.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:26:35.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:26:35.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:612: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:26:35.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2024-08-19T14:26:35.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-08-19T14:26:35.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:26:35.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:26:35.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:26:35.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T14:26:35.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:26:35.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:26:35.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:26:35.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:26:35.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:26:35.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:26:35.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:26:35.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:26:35.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:26:35.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:26:35.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:26:35.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:26:35.333 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:26:35.333 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:26:35.333 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:26:35.333 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:26:35.333 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:26:35.333 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:26:35.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:26:35.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:26:35.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:26:35.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:26:35.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:26:35.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:26:35.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:26:35.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:26:35.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:26:35.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:26:35.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:26:35.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:26:35.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:26:35.334 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:26:35.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=79e74322-123a-47e8-9328-5edf18503e67 2024-08-19T14:26:35.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 79e74322-123a-47e8-9328-5edf18503e67' 2024-08-19T14:26:35.335 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 79e74322-123a-47e8-9328-5edf18503e67 2024-08-19T14:26:35.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:26:35.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAbVsNmoGTTFBAAIb9AoMquDXc+G/3859ma4Q== 2024-08-19T14:26:35.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAbVsNmoGTTFBAAIb9AoMquDXc+G/3859ma4Q=="}' 2024-08-19T14:26:35.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 79e74322-123a-47e8-9328-5edf18503e67 -i td/osd-scrub-repair/0/new.json 2024-08-19T14:26:35.555 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:26:35.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T14:26:35.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAbVsNmoGTTFBAAIb9AoMquDXc+G/3859ma4Q== --osd-uuid 79e74322-123a-47e8-9328-5edf18503e67 2024-08-19T14:26:35.596 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:35.595+0000 7f57c1c11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:26:35.605 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:35.605+0000 7f57c1c11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:26:35.606 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:35.606+0000 7f57c1c11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:26:35.607 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:35.607+0000 7f57c1c11640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:26:35.607 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:35.607+0000 7f57c1c11640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:26:35.607 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:35.607+0000 7f57c1c11640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:26:35.689 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:35.689+0000 7f57c1c11640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T14:26:41.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T14:26:41.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:26:41.074 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T14:26:41.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T14:26:41.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:26:41.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T14:26:41.448 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T14:26:41.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:26:41.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:26:41.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:26:41.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:26:41.476 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:41.476+0000 7f2c807c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:26:41.496 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:41.496+0000 7f2c807c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:26:41.497 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:41.497+0000 7f2c807c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:26:41.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T14:26:41.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:26:41.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:26:41.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:26:41.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:26:41.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:41.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:26:41.744 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:26:41.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:41.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:26:42.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:26:42.030 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:42.030+0000 7f2c807c0640 -1 Falling back to public interface 2024-08-19T14:26:43.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:26:43.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:43.014 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:26:43.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:26:43.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:43.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:26:43.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:26:44.279 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:26:44.279+0000 7f2c807c0640 -1 osd.0 0 log_to_monitors true 2024-08-19T14:26:44.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:26:44.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:44.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:26:44.288 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:26:44.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:44.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:26:44.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:26:45.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:26:45.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:45.556 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:26:45.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:26:45.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:45.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:26:45.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:26:46.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:26:46.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:46.833 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:26:46.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:26:46.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:46.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:26:47.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:26:48.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:26:48.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:48.225 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:26:48.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:26:48.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:48.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:26:48.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:26:49.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:26:49.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:49.504 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:26:49.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:26:49.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:49.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:26:49.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:26:50.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:26:50.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:50.795 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:26:50.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:26:50.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:50.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:26:51.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:26:52.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:26:52.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:52.156 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:26:52.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:26:52.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:52.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:26:52.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:26:53.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:26:53.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:53.433 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:26:53.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:26:53.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:53.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:26:53.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:26:54.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:26:54.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:54.708 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:26:54.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:26:54.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:54.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:26:55.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:26:56.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:26:56.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:56.427 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:26:56.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:26:56.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:56.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:26:57.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:26:58.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:26:58.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:58.023 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:26:58.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:26:58.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:58.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:26:58.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:26:59.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:26:59.514 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:26:59.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:26:59.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:26:59.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:26:59.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:27:00.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:01.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:01.192 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:27:01.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:01.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:27:01.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:01.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:27:01.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:02.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:02.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:02.785 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:27:02.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:27:02.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:02.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:27:03.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:04.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:04.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:04.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:27:04.231 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:27:04.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:04.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:27:04.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:05.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:05.941 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:27:05.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:05.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:27:05.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:05.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:27:06.213 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2175613784,v1:127.0.0.1:6803/2175613784] [v2:127.0.0.1:6804/2175613784,v1:127.0.0.1:6805/2175613784] exists,up 79e74322-123a-47e8-9328-5edf18503e67 2024-08-19T14:27:06.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:27:06.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:27:06.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:27:06.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-08-19T14:27:06.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:27:06.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:27:06.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:27:06.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T14:27:06.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:27:06.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:27:06.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:27:06.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:27:06.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:27:06.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:27:06.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:27:06.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:27:06.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:27:06.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:27:06.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:27:06.218 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:27:06.218 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:27:06.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:27:06.219 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:27:06.219 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:27:06.219 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:27:06.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:27:06.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:27:06.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:27:06.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:27:06.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:27:06.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:27:06.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:27:06.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:27:06.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:27:06.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:27:06.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:27:06.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:27:06.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:27:06.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:27:06.220 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:27:06.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3c8e7feb-cc99-49d1-b41e-762fa1ed0d86 2024-08-19T14:27:06.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3c8e7feb-cc99-49d1-b41e-762fa1ed0d86' 2024-08-19T14:27:06.220 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 3c8e7feb-cc99-49d1-b41e-762fa1ed0d86 2024-08-19T14:27:06.221 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:27:06.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA6VsNmpsL9DRAAI76DUopnijFPLg1cp4NgIQ== 2024-08-19T14:27:06.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA6VsNmpsL9DRAAI76DUopnijFPLg1cp4NgIQ=="}' 2024-08-19T14:27:06.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3c8e7feb-cc99-49d1-b41e-762fa1ed0d86 -i td/osd-scrub-repair/1/new.json 2024-08-19T14:27:06.675 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:27:06.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T14:27:06.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQA6VsNmpsL9DRAAI76DUopnijFPLg1cp4NgIQ== --osd-uuid 3c8e7feb-cc99-49d1-b41e-762fa1ed0d86 2024-08-19T14:27:06.718 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:06.717+0000 7fe144230640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:27:06.751 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:06.750+0000 7fe144230640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:27:06.752 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:06.751+0000 7fe144230640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:27:06.753 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:06.751+0000 7fe144230640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:27:06.753 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:06.751+0000 7fe144230640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:27:06.753 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:06.751+0000 7fe144230640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:27:06.818 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:06.817+0000 7fe144230640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T14:27:12.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T14:27:12.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:27:12.717 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T14:27:12.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T14:27:12.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:27:13.071 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T14:27:13.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T14:27:13.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:27:13.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:27:13.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:27:13.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:27:13.100 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:13.099+0000 7f57b7bd8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:27:13.101 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:13.101+0000 7f57b7bd8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:27:13.102 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:13.102+0000 7f57b7bd8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:27:13.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T14:27:13.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:27:13.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:27:13.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:27:13.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:27:13.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:13.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:27:13.340 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:27:13.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:13.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:27:13.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:13.626 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:13.626+0000 7f57b7bd8640 -1 Falling back to public interface 2024-08-19T14:27:14.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:14.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:14.624 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:27:14.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:27:14.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:14.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:27:14.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:15.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:15.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:15.898 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:27:15.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:27:15.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:15.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:27:16.011 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:16.010+0000 7f57b7bd8640 -1 osd.1 0 log_to_monitors true 2024-08-19T14:27:16.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:17.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:17.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:17.218 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:27:17.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:27:17.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:17.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:27:17.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:18.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:18.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:18.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:27:18.498 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:27:18.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:18.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:27:18.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:19.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:19.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:19.836 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:27:19.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:27:19.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:19.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:27:20.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:21.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:21.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:21.203 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:27:21.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:27:21.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:21.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:27:21.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:22.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:22.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:22.508 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:27:22.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:27:22.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:22.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:27:22.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:23.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:23.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:23.784 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:27:23.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:27:23.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:23.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:27:24.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:25.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:25.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:25.161 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:27:25.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:27:25.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:25.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:27:25.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:26.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:26.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:26.484 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:27:26.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:27:26.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:26.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:27:26.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:27.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:27.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:27.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:27:27.838 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:27:27.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:27.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:27:28.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:29.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:29.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:29.440 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:27:29.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:27:29.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:29.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:27:30.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:31.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:31.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:31.099 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:27:31.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:27:31.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:31.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:27:31.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:32.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:32.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:32.903 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:27:32.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:27:32.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:32.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:27:33.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:34.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:34.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:34.447 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:27:34.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:27:34.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:34.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:27:34.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:35.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:35.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:35.818 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:27:35.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:27:35.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:35.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:27:36.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:37.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:37.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:37.240 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:27:37.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:27:37.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:37.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:27:38.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:39.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:39.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:39.128 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:27:39.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:27:39.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:39.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:27:39.404 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3596589878,v1:127.0.0.1:6811/3596589878] [v2:127.0.0.1:6812/3596589878,v1:127.0.0.1:6813/3596589878] exists,up 3c8e7feb-cc99-49d1-b41e-762fa1ed0d86 2024-08-19T14:27:39.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:27:39.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:27:39.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:27:39.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-08-19T14:27:39.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:27:39.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:27:39.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:27:39.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T14:27:39.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:27:39.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T14:27:39.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:27:39.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:27:39.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:27:39.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:27:39.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T14:27:39.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T14:27:39.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:27:39.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:27:39.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:27:39.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:27:39.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:27:39.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:27:39.410 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:27:39.410 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:27:39.411 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:27:39.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:27:39.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:27:39.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:27:39.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:27:39.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:27:39.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:27:39.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:27:39.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:27:39.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:27:39.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:27:39.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:27:39.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:27:39.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:27:39.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T14:27:39.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:27:39.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=675bdb3c-fab5-4708-96a0-24fe153e0381 2024-08-19T14:27:39.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 675bdb3c-fab5-4708-96a0-24fe153e0381' 2024-08-19T14:27:39.412 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 675bdb3c-fab5-4708-96a0-24fe153e0381 2024-08-19T14:27:39.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:27:39.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBbVsNmEKFhGRAAeF8XFlSuSSFCzGdLbpq4Rw== 2024-08-19T14:27:39.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBbVsNmEKFhGRAAeF8XFlSuSSFCzGdLbpq4Rw=="}' 2024-08-19T14:27:39.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 675bdb3c-fab5-4708-96a0-24fe153e0381 -i td/osd-scrub-repair/2/new.json 2024-08-19T14:27:39.812 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:27:39.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-08-19T14:27:39.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBbVsNmEKFhGRAAeF8XFlSuSSFCzGdLbpq4Rw== --osd-uuid 675bdb3c-fab5-4708-96a0-24fe153e0381 2024-08-19T14:27:39.855 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:39.854+0000 7f5aa413b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:27:39.888 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:39.888+0000 7f5aa413b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:27:39.889 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:39.889+0000 7f5aa413b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:27:39.890 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:39.889+0000 7f5aa413b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:27:39.890 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:39.889+0000 7f5aa413b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:27:39.890 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:39.889+0000 7f5aa413b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:27:39.963 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:39.963+0000 7f5aa413b640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-08-19T14:27:45.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-08-19T14:27:45.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:27:45.607 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T14:27:45.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T14:27:45.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:27:45.957 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T14:27:45.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T14:27:45.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:27:45.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:27:45.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:27:45.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:27:45.986 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:45.985+0000 7fd48e121640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:27:45.988 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:45.988+0000 7fd48e121640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:27:45.989 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:45.989+0000 7fd48e121640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:27:46.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T14:27:46.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:27:46.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T14:27:46.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:27:46.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:27:46.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:46.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:27:46.224 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:27:46.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:46.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:27:46.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:46.512 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:46.512+0000 7fd48e121640 -1 Falling back to public interface 2024-08-19T14:27:47.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:47.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:47.489 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:27:47.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:27:47.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:47.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:27:47.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:48.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:48.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:48.767 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:27:48.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:27:48.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:48.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:27:48.837 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:27:48.837+0000 7fd48e121640 -1 osd.2 0 log_to_monitors true 2024-08-19T14:27:49.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:50.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:50.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:50.055 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:27:50.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:27:50.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:50.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:27:50.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:51.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:51.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:51.333 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:27:51.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:27:51.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:51.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:27:51.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:52.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:52.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:52.643 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:27:52.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:27:52.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:52.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:27:52.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:53.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:53.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:53.920 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:27:53.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:27:53.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:53.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:27:54.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:55.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:55.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:55.198 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:27:55.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:27:55.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:55.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:27:55.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:56.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:56.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:56.498 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:27:56.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:27:56.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:56.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:27:56.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:57.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:57.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:57.882 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:27:57.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:27:57.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:57.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:27:58.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:27:59.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:27:59.161 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:27:59.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:27:59.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:27:59.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:27:59.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:27:59.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:00.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:00.475 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:28:00.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:00.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:28:00.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:00.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:28:00.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:01.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:01.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:01.896 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:28:01.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:28:01.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:01.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:28:02.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:03.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:03.492 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:28:03.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:03.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:28:03.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:03.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:28:04.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:05.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:05.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:05.094 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:28:05.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:28:05.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:05.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:28:05.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:06.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:06.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:06.628 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:28:06.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:28:06.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:06.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:28:07.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:08.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:08.237 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:28:08.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:08.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:28:08.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:08.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:28:08.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:09.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:09.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:09.549 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:28:09.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:28:09.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:09.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:28:10.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:11.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:11.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:11.187 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:28:11.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:28:11.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:11.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:28:11.475 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1839289362,v1:127.0.0.1:6819/1839289362] [v2:127.0.0.1:6820/1839289362,v1:127.0.0.1:6821/1839289362] exists,up 675bdb3c-fab5-4708-96a0-24fe153e0381 2024-08-19T14:28:11.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:28:11.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:28:11.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:28:11.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:617: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2024-08-19T14:28:11.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-08-19T14:28:11.847 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'testpool' created 2024-08-19T14:28:11.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T14:28:12.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2024-08-19T14:28:13.422 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 2 2024-08-19T14:28:13.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:619: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-08-19T14:28:13.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:28:13.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:28:13.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:28:13.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:28:13.434 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:28:13.434 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:28:13.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:28:13.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:28:13.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:28:13.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:28:13.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:28:13.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:28:13.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:28:13.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:28:13.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:28:13.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:28:13.933 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:28:13.933 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:28:13.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:28:13.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:28:13.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:28:14.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-08-19T14:28:14.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-08-19T14:28:14.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-08-19T14:28:14.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:28:14.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:28:14.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-08-19T14:28:14.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-08-19T14:28:14.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672974' 2024-08-19T14:28:14.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:28:14.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:28:14.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-08-19T14:28:14.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-08-19T14:28:14.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672974 2-64424509447' 2024-08-19T14:28:14.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:28:14.308 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-08-19T14:28:14.308 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:28:14.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:28:14.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-08-19T14:28:14.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:28:14.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-08-19T14:28:14.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-08-19T14:28:14.311 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836497 2024-08-19T14:28:14.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:28:14.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-08-19T14:28:14.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:28:15.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:28:15.579 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:28:15.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-08-19T14:28:15.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:28:15.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-08-19T14:28:15.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:28:15.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:28:15.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-08-19T14:28:15.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:28:15.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-08-19T14:28:15.854 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672974 2024-08-19T14:28:15.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-08-19T14:28:15.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:28:16.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-08-19T14:28:16.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:28:16.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-08-19T14:28:16.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:28:16.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:28:16.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-08-19T14:28:16.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:28:16.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-08-19T14:28:16.125 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509447 2024-08-19T14:28:16.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-08-19T14:28:16.126 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:28:16.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-08-19T14:28:16.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:28:16.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:28:16.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:28:16.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:28:16.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:28:16.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:28:16.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:28:16.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:28:16.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:28:16.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:28:16.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:28:16.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:28:16.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:28:16.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:28:16.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:28:17.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:28:17.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:28:17.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:28:17.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:622: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2024-08-19T14:28:17.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2024-08-19T14:28:17.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2024-08-19T14:28:17.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-08-19T14:28:17.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:28:17.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-08-19T14:28:17.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:28:17.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-08-19T14:28:17.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:28:17.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-08-19T14:28:17.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:28:17.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-08-19T14:28:17.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:28:17.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-08-19T14:28:17.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:28:18.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-08-19T14:28:18.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:28:18.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-08-19T14:28:18.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:28:18.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-08-19T14:28:18.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:28:18.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-08-19T14:28:18.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-08-19T14:28:18.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-08-19T14:28:18.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-08-19T14:28:18.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-08-19T14:28:18.542 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-08-19T14:28:18.542 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:28:18.542 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:28:18.542 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:28:18.543 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:28:18.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T14:28:18.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:28:18.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T14:28:19.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-08-19T14:28:19.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:28:19.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:28:19.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T14:28:19.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:28:19.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-08-19T14:28:19.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:28:19.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:28:19.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T14:28:19.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:28:19.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T14:28:19.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:28:19.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:28:19.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:28:19.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:28:19.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:28:19.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:28:19.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-08-19T14:28:19.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:28:19.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:28:19.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T14:28:19.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:28:19.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:28:19.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-08-19T14:28:20.635 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:ff7b1f36:::obj1:head# 2024-08-19T14:28:21.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:28:21.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:28:21.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:28:21.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T14:28:21.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:28:21.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:28:21.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:28:21.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:28:21.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:28:21.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:28:21.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:28:21.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:28:21.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:28:21.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:28:21.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:28:21.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:28:21.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:28:21.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:28:21.154 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:28:21.154 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:28:21.154 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:28:21.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:28:21.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:28:21.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:28:21.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:28:21.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:28:21.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:28:21.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:28:21.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:28:21.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:28:21.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:28:21.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:28:21.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:28:21.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T14:28:21.155 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T14:28:21.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:28:21.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T14:28:21.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T14:28:21.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:28:21.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:28:21.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:28:21.183 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:21.183+0000 7f050d81c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:28:21.185 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:21.185+0000 7f050d81c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:28:21.187 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:21.186+0000 7f050d81c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:28:21.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T14:28:21.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:28:21.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:28:21.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:28:21.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:28:21.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:21.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:28:21.489 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:28:21.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:21.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:28:21.711 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:21.711+0000 7f050d81c640 -1 Falling back to public interface 2024-08-19T14:28:21.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:22.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:22.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:22.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:28:22.763 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:28:22.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:22.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:28:23.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:24.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:24.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:24.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:28:24.040 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:28:24.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:24.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:28:24.064 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:24.063+0000 7f050d81c640 -1 osd.0 21 log_to_monitors true 2024-08-19T14:28:24.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:25.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:25.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:25.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:28:25.394 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:28:25.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:25.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:28:25.672 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/632965030,v1:127.0.0.1:6803/632965030] [v2:127.0.0.1:6804/632965030,v1:127.0.0.1:6805/632965030] exists,up 79e74322-123a-47e8-9328-5edf18503e67 2024-08-19T14:28:25.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:28:25.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:28:25.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:28:25.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:28:25.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:28:25.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:28:25.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:28:25.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:28:25.673 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:28:25.673 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:28:25.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:28:25.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:28:25.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:28:25.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:28:25.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:28:25.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:28:25.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:28:25.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:28:25.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:28:26.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:28:26.024 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:28:26.024 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:28:26.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:28:26.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:28:26.024 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:28:26.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-08-19T14:28:26.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-08-19T14:28:26.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-08-19T14:28:26.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:28:26.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:28:26.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-08-19T14:28:26.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-08-19T14:28:26.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672977' 2024-08-19T14:28:26.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:28:26.228 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:28:26.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-08-19T14:28:26.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-08-19T14:28:26.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672977 2-64424509450' 2024-08-19T14:28:26.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:28:26.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-08-19T14:28:26.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:28:26.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:28:26.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-08-19T14:28:26.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:28:26.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-08-19T14:28:26.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-08-19T14:28:26.333 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 107374182402 2024-08-19T14:28:26.333 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:28:26.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-08-19T14:28:26.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:28:27.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:28:27.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:28:27.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-08-19T14:28:27.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:28:27.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-08-19T14:28:27.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:28:27.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:28:27.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-08-19T14:28:27.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:28:27.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-08-19T14:28:27.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-08-19T14:28:27.877 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949672977 2024-08-19T14:28:27.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:28:28.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-08-19T14:28:28.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:28:28.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-08-19T14:28:28.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:28:28.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:28:28.146 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-08-19T14:28:28.146 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:28:28.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-08-19T14:28:28.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-08-19T14:28:28.148 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509450 2024-08-19T14:28:28.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:28:28.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509450 2024-08-19T14:28:28.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:28:28.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:28:28.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:28:28.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:28:28.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:28:28.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:28:28.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:28:28.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:28:28.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:28:28.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:28:28.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:28:29.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:28:29.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:28:29.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:28:29.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:28:29.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:28:29.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:28:29.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:28:29.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-08-19T14:28:29.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:28:29.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:28:29.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:28:29.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:28:29.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2024-08-19T14:28:29.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:28:29.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:28:29.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T14:28:29.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:28:29.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2024-08-19T14:28:29.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:28:29.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:28:29.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T14:28:29.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:28:29.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T14:28:29.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:28:29.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:28:29.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:28:29.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:28:29.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:28:29.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:28:29.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2024-08-19T14:28:29.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:28:29.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:28:29.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T14:28:29.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:28:29.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:28:29.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2024-08-19T14:28:31.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:28:31.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:28:31.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:28:31.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:28:31.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:28:31.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:28:31.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:28:31.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:28:31.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:28:31.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:28:31.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:28:31.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:28:31.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:28:31.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:28:31.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:28:31.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:28:31.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:28:31.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:28:31.684 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:28:31.684 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:28:31.684 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:28:31.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:28:31.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:28:31.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:28:31.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:28:31.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:28:31.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:28:31.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:28:31.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:28:31.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:28:31.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:28:31.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:28:31.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:28:31.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:28:31.685 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T14:28:31.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:28:31.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T14:28:31.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:28:31.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:28:31.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:28:31.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:28:31.715 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:31.714+0000 7f8d92811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:28:31.717 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:31.717+0000 7f8d92811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:28:31.718 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:31.718+0000 7f8d92811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:28:31.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:28:31.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:28:31.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:28:31.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:28:31.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:28:31.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:31.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:28:31.960 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:28:31.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:31.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:28:32.231 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:32.231+0000 7f8d92811640 -1 Falling back to public interface 2024-08-19T14:28:32.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:33.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:33.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:33.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:28:33.270 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:28:33.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:33.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:28:33.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:34.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:34.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:34.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:28:34.543 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:28:34.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:34.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:28:34.557 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:34.557+0000 7f8d92811640 -1 osd.1 26 log_to_monitors true 2024-08-19T14:28:34.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:35.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:35.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:35.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:28:35.864 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:28:35.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:35.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:28:36.139 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/3183103880,v1:127.0.0.1:6811/3183103880] [v2:127.0.0.1:6812/3183103880,v1:127.0.0.1:6813/3183103880] exists,up 3c8e7feb-cc99-49d1-b41e-762fa1ed0d86 2024-08-19T14:28:36.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:28:36.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:28:36.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:28:36.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:28:36.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:28:36.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:28:36.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:28:36.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:28:36.140 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:28:36.140 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:28:36.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:28:36.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:28:36.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:28:36.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:28:36.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:28:36.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:28:36.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:28:36.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:28:36.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:28:36.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:28:36.494 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:28:36.494 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:28:36.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:28:36.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:28:36.494 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:28:36.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182405 2024-08-19T14:28:36.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182405 2024-08-19T14:28:36.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405' 2024-08-19T14:28:36.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:28:36.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:28:36.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-08-19T14:28:36.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-08-19T14:28:36.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-128849018882' 2024-08-19T14:28:36.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:28:36.701 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:28:36.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-08-19T14:28:36.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-08-19T14:28:36.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-128849018882 2-64424509454' 2024-08-19T14:28:36.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:28:36.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182405 2024-08-19T14:28:36.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:28:36.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:28:36.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182405 2024-08-19T14:28:36.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:28:36.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182405 2024-08-19T14:28:36.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-08-19T14:28:36.805 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 107374182405 2024-08-19T14:28:36.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:28:37.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182405 2024-08-19T14:28:37.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:28:38.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:28:38.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:28:38.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182405 2024-08-19T14:28:38.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:28:38.344 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018882 2024-08-19T14:28:38.344 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:28:38.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:28:38.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018882 2024-08-19T14:28:38.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:28:38.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-08-19T14:28:38.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-08-19T14:28:38.347 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 128849018882 2024-08-19T14:28:38.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:28:38.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-08-19T14:28:38.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:28:38.610 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-08-19T14:28:38.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:28:38.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:28:38.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-08-19T14:28:38.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:28:38.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-08-19T14:28:38.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-08-19T14:28:38.613 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509454 2024-08-19T14:28:38.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:28:38.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-08-19T14:28:38.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:28:38.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:28:38.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:28:39.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:28:39.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:28:39.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:28:39.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:28:39.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:28:39.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:28:39.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:28:39.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:28:39.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:28:39.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:28:39.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:28:39.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:28:39.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:28:39.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:28:39.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:28:39.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-08-19T14:28:39.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-08-19T14:28:39.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-08-19T14:28:39.898 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-08-19T14:28:39.898 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:28:39.898 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:28:39.898 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:28:39.898 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:28:40.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T14:28:40.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:28:40.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T14:28:40.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-08-19T14:28:40.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:28:40.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:28:40.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T14:28:40.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:28:40.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-08-19T14:28:40.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:28:40.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:28:40.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T14:28:40.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:28:40.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T14:28:40.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:28:40.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:28:40.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:28:40.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:28:40.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:28:40.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:28:40.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-08-19T14:28:40.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:28:40.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:28:40.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T14:28:40.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:28:40.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:28:40.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-08-19T14:28:42.071 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:104778fc:::obj2:head# 2024-08-19T14:28:42.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:28:42.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:28:42.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:28:42.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T14:28:42.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:28:42.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:28:42.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:28:42.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:28:42.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:28:42.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:28:42.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:28:42.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:28:42.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:28:42.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:28:42.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:28:42.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:28:42.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:28:42.590 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:28:42.590 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:28:42.590 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:28:42.590 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:28:42.590 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:28:42.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:28:42.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:28:42.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:28:42.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:28:42.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:28:42.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:28:42.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:28:42.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:28:42.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:28:42.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:28:42.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:28:42.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T14:28:42.591 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T14:28:42.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:28:42.592 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T14:28:42.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T14:28:42.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:28:42.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:28:42.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:28:42.621 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:42.619+0000 7ffae080d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:28:42.623 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:42.621+0000 7ffae080d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:28:42.624 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:42.623+0000 7ffae080d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:28:42.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T14:28:42.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:28:42.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:28:42.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:28:42.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:28:42.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:42.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:28:42.867 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:28:42.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:42.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:28:43.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:43.144 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:43.143+0000 7ffae080d640 -1 Falling back to public interface 2024-08-19T14:28:44.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:44.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:44.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:28:44.133 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:28:44.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:44.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:28:44.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:45.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:45.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:45.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:28:45.411 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:28:45.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:45.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:28:45.580 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:45.578+0000 7ffae080d640 -1 osd.0 31 log_to_monitors true 2024-08-19T14:28:45.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:46.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:46.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:46.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:28:46.717 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:28:46.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:46.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:28:46.991 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 35 up_thru 27 down_at 32 last_clean_interval [25,31) [v2:127.0.0.1:6802/2138768446,v1:127.0.0.1:6803/2138768446] [v2:127.0.0.1:6804/2138768446,v1:127.0.0.1:6805/2138768446] exists,up 79e74322-123a-47e8-9328-5edf18503e67 2024-08-19T14:28:46.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:28:46.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:28:46.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:28:46.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:28:46.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:28:46.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:28:46.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:28:46.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:28:46.992 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:28:46.992 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:28:46.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:28:46.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:28:46.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:28:47.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:28:47.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:28:47.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:28:47.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:28:47.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:28:47.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:28:47.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:28:47.346 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:28:47.346 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:28:47.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:28:47.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:28:47.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:28:47.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-08-19T14:28:47.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-08-19T14:28:47.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-08-19T14:28:47.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:28:47.449 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:28:47.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018885 2024-08-19T14:28:47.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018885 2024-08-19T14:28:47.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-128849018885' 2024-08-19T14:28:47.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:28:47.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:28:47.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-08-19T14:28:47.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-08-19T14:28:47.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-128849018885 2-64424509457' 2024-08-19T14:28:47.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:28:47.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-08-19T14:28:47.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:28:47.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:28:47.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-08-19T14:28:47.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:28:47.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-08-19T14:28:47.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-08-19T14:28:47.655 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 150323855362 2024-08-19T14:28:47.655 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:28:47.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-08-19T14:28:47.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:28:48.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:28:48.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:28:49.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-08-19T14:28:49.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:28:50.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T14:28:50.195 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:28:50.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-08-19T14:28:50.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:28:50.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018885 2024-08-19T14:28:50.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:28:50.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:28:50.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018885 2024-08-19T14:28:50.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:28:50.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018885 2024-08-19T14:28:50.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018885' 2024-08-19T14:28:50.474 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 128849018885 2024-08-19T14:28:50.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:28:50.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018885 -lt 128849018885 2024-08-19T14:28:50.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:28:50.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-08-19T14:28:50.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:28:50.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:28:50.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-08-19T14:28:50.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:28:50.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-08-19T14:28:50.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-08-19T14:28:50.745 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509457 2024-08-19T14:28:50.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:28:51.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-08-19T14:28:51.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:28:51.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:28:51.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:28:51.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:28:51.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:28:51.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:28:51.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:28:51.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:28:51.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:28:51.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:28:51.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:28:51.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:28:51.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:28:51.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:28:51.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:28:51.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:28:51.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:28:51.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:28:51.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-08-19T14:28:51.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:28:51.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:28:51.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:28:51.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:28:52.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-08-19T14:28:52.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:28:52.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:28:52.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T14:28:52.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:28:52.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-08-19T14:28:52.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:28:52.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:28:52.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T14:28:52.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:28:52.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T14:28:52.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:28:52.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:28:52.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:28:52.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:28:52.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:28:52.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:28:52.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-08-19T14:28:52.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:28:52.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:28:52.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T14:28:52.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:28:52.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:28:52.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-08-19T14:28:53.743 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2024-08-19T14:28:54.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:28:54.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:28:54.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:28:54.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:28:54.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:28:54.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:28:54.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:28:54.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:28:54.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:28:54.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:28:54.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:28:54.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:28:54.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:28:54.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:28:54.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:28:54.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:28:54.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:28:54.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:28:54.263 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:28:54.263 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:28:54.264 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:28:54.264 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:28:54.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:28:54.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:28:54.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:28:54.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:28:54.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:28:54.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:28:54.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:28:54.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:28:54.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:28:54.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:28:54.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:28:54.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:28:54.265 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T14:28:54.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:28:54.265 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T14:28:54.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:28:54.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:28:54.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:28:54.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:28:54.294 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:54.293+0000 7febfd21a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:28:54.296 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:54.296+0000 7febfd21a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:28:54.298 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:54.297+0000 7febfd21a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:28:54.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:28:54.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:28:54.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:28:54.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:28:54.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:28:54.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:54.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:28:54.541 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:28:54.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:54.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:28:54.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:54.825 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:54.825+0000 7febfd21a640 -1 Falling back to public interface 2024-08-19T14:28:55.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:55.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:55.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:28:55.806 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:28:55.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:55.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:28:56.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:57.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:57.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:57.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:28:57.087 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:28:57.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:57.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:28:57.115 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:28:57.115+0000 7febfd21a640 -1 osd.1 36 log_to_monitors true 2024-08-19T14:28:57.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:28:58.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:28:58.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:28:58.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:28:58.429 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:28:58.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:28:58.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:28:58.701 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6810/3761980786,v1:127.0.0.1:6811/3761980786] [v2:127.0.0.1:6812/3761980786,v1:127.0.0.1:6813/3761980786] exists,up 3c8e7feb-cc99-49d1-b41e-762fa1ed0d86 2024-08-19T14:28:58.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:28:58.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:28:58.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:28:58.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:28:58.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:28:58.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:28:58.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:28:58.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:28:58.702 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:28:58.703 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:28:58.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:28:58.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:28:58.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:28:58.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:28:58.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:28:58.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:28:58.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:28:58.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:28:58.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:28:59.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:28:59.056 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:28:59.056 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:28:59.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:28:59.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:28:59.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:28:59.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-08-19T14:28:59.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-08-19T14:28:59.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365' 2024-08-19T14:28:59.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:28:59.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:28:59.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-08-19T14:28:59.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-08-19T14:28:59.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365 1-171798691842' 2024-08-19T14:28:59.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:28:59.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:28:59.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-08-19T14:28:59.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-08-19T14:28:59.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365 1-171798691842 2-64424509460' 2024-08-19T14:28:59.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:28:59.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855365 2024-08-19T14:28:59.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:28:59.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:28:59.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855365 2024-08-19T14:28:59.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:28:59.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-08-19T14:28:59.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855365' 2024-08-19T14:28:59.368 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 150323855365 2024-08-19T14:28:59.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:28:59.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-08-19T14:28:59.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:28:59.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691842 2024-08-19T14:28:59.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:28:59.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:28:59.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691842 2024-08-19T14:28:59.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:28:59.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-08-19T14:28:59.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-08-19T14:28:59.637 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 171798691842 2024-08-19T14:28:59.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:28:59.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 171798691842 2024-08-19T14:28:59.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:29:00.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:29:00.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:29:01.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 171798691842 2024-08-19T14:29:01.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:29:02.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T14:29:02.180 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:29:02.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-08-19T14:29:02.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:29:02.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-08-19T14:29:02.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:29:02.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:29:02.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-08-19T14:29:02.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:29:02.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-08-19T14:29:02.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-08-19T14:29:02.455 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509460 2024-08-19T14:29:02.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:29:02.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-08-19T14:29:02.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:29:02.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:29:02.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:29:03.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:29:03.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:29:03.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:29:03.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:29:03.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:29:03.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:29:03.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:29:03.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:29:03.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:29:03.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:29:03.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:29:03.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:29:03.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:29:03.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:29:03.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:29:03.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-08-19T14:29:03.767 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: { 2024-08-19T14:29:03.767 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-08-19T14:29:03.767 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:29:03.781 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: { 2024-08-19T14:29:03.781 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-08-19T14:29:03.781 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:29:03.793 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: { 2024-08-19T14:29:03.793 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-08-19T14:29:03.794 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:29:03.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2024-08-19T14:29:03.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-08-19T14:29:03.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-08-19T14:29:03.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-08-19T14:29:03.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:29:04.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2024-08-19T14:29:04.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2024-08-19T14:29:04.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:29:04.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-08-19T14:29:04.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-08-19T14:29:04.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:29:04.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2024-08-19T14:29:04.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2024-08-19T14:29:04.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:29:04.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:29:04.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:29:04.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:29:04.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2024-08-19T14:28:11.795397+0000 2024-08-19T14:29:04.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 schedule-deep-scrub 2024-08-19T14:29:04.704 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T14:29:04.704 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": true, 2024-08-19T14:29:04.704 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T14:29:04.704 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-05T14:27:24.704403+0000" 2024-08-19T14:29:04.704 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:29:04.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:645: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2024-08-19T14:28:11.795397+0000 2024-08-19T14:29:04.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-08-19T14:29:04.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:28:11.795397+0000 2024-08-19T14:29:04.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:29:04.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:29:04.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:29:04.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:29:04.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:29:04.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:29:04.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:29:04.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:29:04.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:28:11.795397+0000 '>' 2024-08-19T14:28:11.795397+0000 2024-08-19T14:29:04.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:29:05.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:29:05.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:29:05.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:29:05.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:29:05.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:29:05.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:29:05.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:29:06.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:28:11.795397+0000 '>' 2024-08-19T14:28:11.795397+0000 2024-08-19T14:29:06.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:29:07.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:29:07.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:29:07.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:29:07.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:29:07.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:29:07.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:29:07.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:29:07.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:28:11.795397+0000 '>' 2024-08-19T14:28:11.795397+0000 2024-08-19T14:29:07.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:29:08.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:29:08.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:29:08.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:29:08.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:29:08.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:29:08.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:29:08.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:29:08.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:28:11.795397+0000 '>' 2024-08-19T14:28:11.795397+0000 2024-08-19T14:29:08.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:29:09.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:29:09.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:29:09.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:29:09.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:29:09.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:29:09.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:29:09.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:29:10.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T14:27:24.704403+0000 '>' 2024-08-19T14:28:11.795397+0000 2024-08-19T14:29:10.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:29:11.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:29:11.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:29:11.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:29:11.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:29:11.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:29:11.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:29:11.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:29:11.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T14:27:24.704403+0000 '>' 2024-08-19T14:28:11.795397+0000 2024-08-19T14:29:11.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:29:12.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:29:12.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:29:12.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:29:12.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:29:12.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:29:12.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:29:12.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:29:12.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T14:27:24.704403+0000 '>' 2024-08-19T14:28:11.795397+0000 2024-08-19T14:29:12.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:29:13.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:29:13.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:29:13.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:29:13.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:29:13.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:29:13.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:29:13.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:29:13.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:29:11.279995+0000 '>' 2024-08-19T14:28:11.795397+0000 2024-08-19T14:29:13.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:29:13.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-08-19T14:29:13.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:29:13.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:29:13.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:29:13.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:29:13.898 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:29:13.898 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:29:13.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:29:13.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:29:13.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:29:13.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:29:13.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:29:13.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:29:13.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:29:13.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:29:13.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:29:14.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:29:14.249 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:29:14.249 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:29:14.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:29:14.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:29:14.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:29:14.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855369 2024-08-19T14:29:14.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855369 2024-08-19T14:29:14.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855369' 2024-08-19T14:29:14.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:29:14.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:29:14.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691846 2024-08-19T14:29:14.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691846 2024-08-19T14:29:14.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855369 1-171798691846' 2024-08-19T14:29:14.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:29:14.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:29:14.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-08-19T14:29:14.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-08-19T14:29:14.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855369 1-171798691846 2-64424509464' 2024-08-19T14:29:14.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:29:14.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855369 2024-08-19T14:29:14.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:29:14.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:29:14.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855369 2024-08-19T14:29:14.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:29:14.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855369 2024-08-19T14:29:14.561 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 150323855369 2024-08-19T14:29:14.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855369' 2024-08-19T14:29:14.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:29:14.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855368 -lt 150323855369 2024-08-19T14:29:14.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:29:15.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:29:15.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:29:16.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855369 -lt 150323855369 2024-08-19T14:29:16.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:29:16.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691846 2024-08-19T14:29:16.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:29:16.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:29:16.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691846 2024-08-19T14:29:16.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:29:16.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691846 2024-08-19T14:29:16.105 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 171798691846 2024-08-19T14:29:16.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691846' 2024-08-19T14:29:16.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:29:16.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691846 -lt 171798691846 2024-08-19T14:29:16.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:29:16.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-08-19T14:29:16.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:29:16.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:29:16.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-08-19T14:29:16.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:29:16.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-08-19T14:29:16.410 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509464 2024-08-19T14:29:16.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-08-19T14:29:16.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:29:16.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-08-19T14:29:16.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:29:16.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:29:16.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:29:17.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:29:17.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:29:17.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:29:17.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:29:17.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:29:17.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:29:17.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:29:17.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:29:17.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:29:17.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:29:17.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:29:17.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:29:17.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:29:17.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:29:17.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:29:17.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2024-08-19T14:29:17.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:29:17.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:29:17.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:29:17.866 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:29:17.866 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:29:17.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:29:17.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:29:17.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:29:17.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855371 2024-08-19T14:29:17.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855371 2024-08-19T14:29:17.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855371' 2024-08-19T14:29:17.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:29:17.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:29:18.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691848 2024-08-19T14:29:18.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691848 2024-08-19T14:29:18.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855371 1-171798691848' 2024-08-19T14:29:18.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:29:18.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:29:18.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509466 2024-08-19T14:29:18.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509466 2024-08-19T14:29:18.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855371 1-171798691848 2-64424509466' 2024-08-19T14:29:18.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:29:18.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855371 2024-08-19T14:29:18.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:29:18.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:29:18.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855371 2024-08-19T14:29:18.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:29:18.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855371 2024-08-19T14:29:18.179 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 150323855371 2024-08-19T14:29:18.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855371' 2024-08-19T14:29:18.180 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:29:18.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855370 -lt 150323855371 2024-08-19T14:29:18.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:29:19.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:29:19.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:29:19.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855371 -lt 150323855371 2024-08-19T14:29:19.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:29:19.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691848 2024-08-19T14:29:19.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:29:19.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:29:19.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691848 2024-08-19T14:29:19.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:29:19.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691848 2024-08-19T14:29:19.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691848' 2024-08-19T14:29:19.719 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 171798691848 2024-08-19T14:29:19.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:29:19.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691848 -lt 171798691848 2024-08-19T14:29:19.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:29:19.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509466 2024-08-19T14:29:19.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:29:19.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:29:19.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509466 2024-08-19T14:29:19.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:29:19.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509466 2024-08-19T14:29:19.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-08-19T14:29:19.991 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509466 2024-08-19T14:29:19.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:29:20.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509466 -lt 64424509466 2024-08-19T14:29:20.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2024-08-19T14:29:20.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-08-19T14:29:20.553 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T14:29:20.554 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-08-19T14:29:11.280103+0000 21'10 41:97 [1,0] 1 [1,0] 1 21'10 2024-08-19T14:29:11.279995+0000 21'10 2024-08-19T14:29:11.279995+0000 0 1 periodic scrub scheduled @ 2024-08-20T14:29:11.279995+0000 10 0 2024-08-19T14:29:20.554 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:29:20.554 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T14:29:20.554 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:29:20.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-08-19T14:29:20.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2024-08-19T14:29:20.813 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:29:20.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T14:29:20.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:29:20.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:29:20.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:29:20.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:29:20.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:29:20.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:29:20.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:29:20.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:29:20.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:29:20.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:29:20.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:29:20.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:29:20.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:29:20.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:29:20.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:29:20.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:29:20.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:29:20.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:29:20.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:29:20.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:29:20.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:29:20.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:29:20.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:29:20.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:29:20.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:29:20.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:29:20.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:29:20.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:29:20.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T14:29:20.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T14:29:20.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T14:29:20.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T14:29:20.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:29:20.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:29:20.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:29:20.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:29:20.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:29:20.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:29:20.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:29:20.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:29:20.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:29:20.959 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:29:20.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:29:20.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:29:20.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:29:20.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:29:20.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:29:20.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:29:20.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:29:20.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:29:20.962 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:29:20.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:29:20.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:29:20.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:29:20.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:29:20.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:29:20.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:29:20.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:29:20.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:29:20.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:29:20.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T14:29:20.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T14:29:20.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:29:20.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:29:20.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T14:29:20.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T14:29:20.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T14:29:20.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T14:29:20.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T14:29:20.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2024-08-19T14:29:20.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2024-08-19T14:29:20.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:484: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2024-08-19T14:29:20.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2024-08-19T14:29:20.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T14:29:20.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T14:29:20.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T14:29:20.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T14:29:20.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T14:29:20.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-08-19T14:29:21.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T14:29:21.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:29:21.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:29:21.315 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:29:21.315 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:29:21.315 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:29:21.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:29:21.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T14:29:21.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T14:29:21.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T14:29:21.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:29:21.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:29:21.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T14:29:21.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T14:29:21.508 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:29:21.508 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:29:21.508 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:29:21.508 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:29:21.509 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:29:21.509 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:29:21.509 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:29:21.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:29:21.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T14:29:21.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T14:29:21.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:29:21.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:29:21.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T14:29:21.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T14:29:21.587 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:29:21.587 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:29:21.587 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:29:21.587 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:29:21.587 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:29:21.588 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:29:21.588 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:29:21.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:29:21.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T14:29:21.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:488: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2024-08-19T14:29:21.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T14:29:21.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T14:29:21.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T14:29:21.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T14:29:21.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T14:29:21.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T14:29:22.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T14:29:22.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:29:22.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:29:22.475 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:29:22.475 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:29:22.476 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:29:22.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:29:22.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:29:22.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:29:22.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-08-19T14:29:22.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: seq 0 2 2024-08-19T14:29:22.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-08-19T14:29:22.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-08-19T14:29:22.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:29:22.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:29:22.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T14:29:22.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:29:22.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:29:22.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:29:22.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:29:22.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:29:22.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:29:22.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:29:22.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:29:22.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:29:22.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:29:22.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:29:22.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:29:22.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:29:22.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:29:22.509 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:29:22.509 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:29:22.509 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:29:22.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:29:22.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:29:22.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:29:22.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:29:22.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:29:22.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:29:22.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:29:22.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:29:22.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:29:22.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:29:22.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:29:22.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:29:22.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-08-19T14:29:22.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:29:22.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:29:22.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4547bd6b-d083-4d49-a062-1439bb29fe29 2024-08-19T14:29:22.511 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 4547bd6b-d083-4d49-a062-1439bb29fe29 2024-08-19T14:29:22.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4547bd6b-d083-4d49-a062-1439bb29fe29' 2024-08-19T14:29:22.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:29:22.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDCVsNmgxpbHxAA6ouxF7LD19Jy3voTakx9IA== 2024-08-19T14:29:22.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDCVsNmgxpbHxAA6ouxF7LD19Jy3voTakx9IA=="}' 2024-08-19T14:29:22.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4547bd6b-d083-4d49-a062-1439bb29fe29 -i td/osd-scrub-repair/0/new.json 2024-08-19T14:29:22.748 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:29:22.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T14:29:22.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDCVsNmgxpbHxAA6ouxF7LD19Jy3voTakx9IA== --osd-uuid 4547bd6b-d083-4d49-a062-1439bb29fe29 2024-08-19T14:29:22.790 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:29:22.789+0000 7faf7ea9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:29:22.798 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:29:22.798+0000 7faf7ea9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:29:22.799 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:29:22.799+0000 7faf7ea9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:29:22.800 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:29:22.800+0000 7faf7ea9c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:29:22.800 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:29:22.800+0000 7faf7ea9c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:29:22.800 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:29:22.800+0000 7faf7ea9c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:29:22.882 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:29:22.882+0000 7faf7ea9c640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T14:29:28.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T14:29:28.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:29:28.230 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T14:29:28.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T14:29:28.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:29:28.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T14:29:28.607 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T14:29:28.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-08-19T14:29:28.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:29:28.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:29:28.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:29:28.636 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:29:28.636+0000 7f67dfd4e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:29:28.664 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:29:28.664+0000 7f67dfd4e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:29:28.665 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:29:28.665+0000 7f67dfd4e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:29:28.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T14:29:28.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:29:28.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:29:28.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:29:28.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:29:28.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:29:28.874 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:29:28.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:29:28.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:29:28.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:29:29.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:29:29.186 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:29:29.186+0000 7f67dfd4e640 -1 Falling back to public interface 2024-08-19T14:29:30.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:29:30.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:29:30.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:29:30.163 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:29:30.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:29:30.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:29:30.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:29:31.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:29:31.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:29:31.438 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:29:31.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:29:31.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:29:31.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:29:31.514 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:29:31.512+0000 7f67dfd4e640 -1 osd.0 0 log_to_monitors true 2024-08-19T14:29:31.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:29:32.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:29:32.781 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:29:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:29:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:29:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:29:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:29:33.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:29:34.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:29:34.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:29:34.149 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:29:34.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:29:34.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:29:34.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:29:34.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:29:35.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:29:35.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:29:35.426 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:29:35.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:29:35.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:29:35.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:29:35.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:29:36.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:29:36.705 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:29:36.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:29:36.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:29:36.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:29:36.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:29:36.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:29:37.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:29:37.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:29:37.986 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:29:37.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:29:37.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:29:37.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:29:38.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:29:39.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:29:39.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:29:39.388 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:29:39.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:29:39.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:29:39.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:29:39.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:29:40.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:29:40.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:29:40.665 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:29:40.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:29:40.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:29:40.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:29:41.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:29:42.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:29:42.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:29:42.005 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:29:42.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:29:42.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:29:42.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:29:42.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:29:43.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:29:43.351 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:29:43.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:29:43.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:29:43.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:29:43.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:29:43.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:29:44.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:29:44.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:29:44.804 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:29:44.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:29:44.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:29:44.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:29:45.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:29:46.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:29:46.320 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:29:46.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:29:46.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:29:46.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:29:46.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:29:47.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:29:48.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:29:48.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:29:48.116 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:29:48.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:29:48.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:29:48.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:29:48.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:29:49.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:29:49.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:29:49.776 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:29:49.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:29:49.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:29:49.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:29:50.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:29:51.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:29:51.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:29:51.471 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:29:51.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:29:51.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:29:51.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:29:52.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:29:53.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:29:53.231 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:29:53.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:29:53.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:29:53.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:29:53.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:29:53.626 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/174115431,v1:127.0.0.1:6803/174115431] [v2:127.0.0.1:6804/174115431,v1:127.0.0.1:6805/174115431] exists,up 4547bd6b-d083-4d49-a062-1439bb29fe29 2024-08-19T14:29:53.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:29:53.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:29:53.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:29:53.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-08-19T14:29:53.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-08-19T14:29:53.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:29:53.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:29:53.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T14:29:53.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:29:53.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:29:53.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:29:53.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:29:53.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:29:53.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:29:53.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:29:53.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:29:53.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:29:53.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:29:53.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:29:53.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:29:53.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:29:53.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:29:53.632 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:29:53.632 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:29:53.632 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:29:53.633 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:29:53.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:29:53.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:29:53.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:29:53.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:29:53.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:29:53.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:29:53.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:29:53.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:29:53.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:29:53.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:29:53.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:29:53.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-08-19T14:29:53.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:29:53.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:29:53.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7f12ac18-9b1c-4557-9643-47af03db872f 2024-08-19T14:29:53.634 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 7f12ac18-9b1c-4557-9643-47af03db872f 2024-08-19T14:29:53.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7f12ac18-9b1c-4557-9643-47af03db872f' 2024-08-19T14:29:53.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:29:53.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDhVsNmWSipJhAAWeqmJntHtMvvnw7CckPAXw== 2024-08-19T14:29:53.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDhVsNmWSipJhAAWeqmJntHtMvvnw7CckPAXw=="}' 2024-08-19T14:29:53.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7f12ac18-9b1c-4557-9643-47af03db872f -i td/osd-scrub-repair/1/new.json 2024-08-19T14:29:53.975 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:29:53.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T14:29:53.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDhVsNmWSipJhAAWeqmJntHtMvvnw7CckPAXw== --osd-uuid 7f12ac18-9b1c-4557-9643-47af03db872f 2024-08-19T14:29:54.018 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:29:54.017+0000 7f0764e5f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:29:54.060 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:29:54.059+0000 7f0764e5f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:29:54.061 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:29:54.060+0000 7f0764e5f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:29:54.062 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:29:54.061+0000 7f0764e5f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:29:54.062 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:29:54.061+0000 7f0764e5f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:29:54.062 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:29:54.061+0000 7f0764e5f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:29:54.127 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:29:54.126+0000 7f0764e5f640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T14:29:59.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T14:29:59.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:29:59.893 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T14:29:59.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T14:29:59.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:30:00.238 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T14:30:00.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T14:30:00.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-08-19T14:30:00.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:30:00.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:30:00.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:30:00.267 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:30:00.267+0000 7f2c3eb14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:30:00.269 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:30:00.269+0000 7f2c3eb14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:30:00.270 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:30:00.270+0000 7f2c3eb14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:30:00.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T14:30:00.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:30:00.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:30:00.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:30:00.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:30:00.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:00.513 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:30:00.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:30:00.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:00.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:30:00.785 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:30:00.785+0000 7f2c3eb14640 -1 Falling back to public interface 2024-08-19T14:30:00.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:01.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:01.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:01.794 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:30:01.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:30:01.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:01.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:30:02.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:03.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:03.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:03.069 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:30:03.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:30:03.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:03.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:30:03.117 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:30:03.117+0000 7f2c3eb14640 -1 osd.1 0 log_to_monitors true 2024-08-19T14:30:03.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:04.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:04.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:04.419 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:30:04.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:30:04.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:04.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:30:04.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:05.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:05.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:05.868 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:30:05.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:30:05.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:05.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:30:06.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:07.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:07.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:07.145 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:30:07.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:30:07.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:07.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:30:07.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:08.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:08.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:08.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:30:08.419 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:30:08.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:08.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:30:08.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:09.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:09.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:09.810 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:30:09.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:30:09.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:09.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:30:10.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:11.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:11.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:11.123 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:30:11.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:30:11.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:11.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:30:11.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:12.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:12.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:12.397 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:30:12.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:30:12.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:12.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:30:12.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:13.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:13.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:13.711 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:30:13.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:30:13.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:13.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:30:14.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:15.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:15.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:15.082 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:30:15.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:30:15.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:15.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:30:15.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:16.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:16.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:16.587 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:30:16.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:30:16.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:16.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:30:17.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:18.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:18.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:18.152 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:30:18.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:30:18.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:18.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:30:18.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:19.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:19.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:19.529 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:30:19.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:30:19.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:19.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:30:20.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:21.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:21.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:21.257 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:30:21.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:30:21.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:21.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:30:21.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:22.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:22.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:22.957 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:30:22.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:30:22.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:22.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:30:23.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:24.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:24.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:24.746 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:30:24.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:30:24.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:24.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:30:25.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:26.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:26.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:26.143 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:30:26.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:30:26.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:26.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:30:26.422 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4027222767,v1:127.0.0.1:6811/4027222767] [v2:127.0.0.1:6812/4027222767,v1:127.0.0.1:6813/4027222767] exists,up 7f12ac18-9b1c-4557-9643-47af03db872f 2024-08-19T14:30:26.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:30:26.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:30:26.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:30:26.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-08-19T14:30:26.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-08-19T14:30:26.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:30:26.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:30:26.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T14:30:26.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:30:26.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T14:30:26.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:30:26.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:30:26.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:30:26.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:30:26.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T14:30:26.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T14:30:26.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:30:26.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:30:26.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:30:26.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:30:26.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:30:26.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:30:26.428 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:30:26.428 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:30:26.428 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:30:26.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:30:26.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:30:26.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:30:26.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:30:26.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:30:26.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:30:26.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:30:26.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:30:26.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:30:26.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:30:26.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:30:26.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:30:26.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-08-19T14:30:26.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T14:30:26.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:30:26.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cd7fadfd-5979-4f38-ad8f-3bb3abf72a66 2024-08-19T14:30:26.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cd7fadfd-5979-4f38-ad8f-3bb3abf72a66' 2024-08-19T14:30:26.430 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 cd7fadfd-5979-4f38-ad8f-3bb3abf72a66 2024-08-19T14:30:26.430 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:30:26.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQACV8NmlohzGhAAvf97HIeQbOHZ8Qe7b0yk6w== 2024-08-19T14:30:26.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQACV8NmlohzGhAAvf97HIeQbOHZ8Qe7b0yk6w=="}' 2024-08-19T14:30:26.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cd7fadfd-5979-4f38-ad8f-3bb3abf72a66 -i td/osd-scrub-repair/2/new.json 2024-08-19T14:30:26.800 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:30:26.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-08-19T14:30:26.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQACV8NmlohzGhAAvf97HIeQbOHZ8Qe7b0yk6w== --osd-uuid cd7fadfd-5979-4f38-ad8f-3bb3abf72a66 2024-08-19T14:30:26.843 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:30:26.842+0000 7fdfff615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:30:26.871 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:30:26.871+0000 7fdfff615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:30:26.872 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:30:26.872+0000 7fdfff615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:30:26.873 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:30:26.872+0000 7fdfff615640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:30:26.873 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:30:26.872+0000 7fdfff615640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:30:26.873 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:30:26.872+0000 7fdfff615640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:30:26.930 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:30:26.929+0000 7fdfff615640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-08-19T14:30:32.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-08-19T14:30:32.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:30:32.374 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T14:30:32.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T14:30:32.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:30:32.747 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T14:30:32.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T14:30:32.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-08-19T14:30:32.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:30:32.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:30:32.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:30:32.776 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:30:32.776+0000 7f0e25a1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:30:32.778 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:30:32.778+0000 7f0e25a1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:30:32.779 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:30:32.779+0000 7f0e25a1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:30:33.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T14:30:33.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:30:33.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T14:30:33.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:30:33.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:30:33.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:33.015 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:30:33.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:30:33.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:33.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:33.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:33.312 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:30:33.312+0000 7f0e25a1e640 -1 Falling back to public interface 2024-08-19T14:30:34.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:34.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:34.280 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:30:34.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:30:34.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:34.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:34.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:35.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:35.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:35.556 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:30:35.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:30:35.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:35.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:35.634 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:30:35.633+0000 7f0e25a1e640 -1 osd.2 0 log_to_monitors true 2024-08-19T14:30:35.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:36.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:36.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:36.843 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:30:36.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:30:36.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:36.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:37.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:38.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:38.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:38.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:30:38.260 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:30:38.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:38.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:38.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:39.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:39.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:39.604 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:30:39.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:30:39.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:39.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:39.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:40.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:40.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:40.879 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:30:40.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:30:40.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:40.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:41.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:42.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:42.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:42.158 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:30:42.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:30:42.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:42.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:42.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:43.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:43.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:43.471 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:30:43.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:30:43.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:43.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:43.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:44.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:44.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:44.768 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:30:44.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:30:44.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:44.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:45.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:46.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:46.148 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:30:46.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:46.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:30:46.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:46.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:46.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:47.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:47.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:47.446 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:30:47.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:30:47.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:47.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:47.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:48.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:48.724 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:30:48.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:48.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:30:48.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:48.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:49.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:50.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:50.372 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:30:50.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:50.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:30:50.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:50.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:50.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:51.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:51.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:51.741 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:30:51.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:30:51.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:51.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:52.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:53.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:53.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:53.208 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:30:53.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:30:53.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:53.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:53.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:54.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:54.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:54.562 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:30:54.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:30:54.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:54.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:54.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:55.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:55.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:55.971 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:30:55.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:30:55.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:55.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:56.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:57.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:57.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:57.652 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:30:57.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:30:57.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:57.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:57.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:30:58.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:30:58.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:30:58.946 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T14:30:58.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T14:30:58.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:30:58.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:30:59.223 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/795411896,v1:127.0.0.1:6819/795411896] [v2:127.0.0.1:6820/795411896,v1:127.0.0.1:6821/795411896] exists,up cd7fadfd-5979-4f38-ad8f-3bb3abf72a66 2024-08-19T14:30:59.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:30:59.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:30:59.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:30:59.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2024-08-19T14:30:59.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-08-19T14:30:59.574 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'testpool' created 2024-08-19T14:30:59.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T14:31:00.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2024-08-19T14:31:01.090 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 2 2024-08-19T14:31:01.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-08-19T14:31:01.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:31:01.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:31:01.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:31:01.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:31:01.105 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:31:01.105 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:31:01.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:31:01.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:31:01.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:31:01.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:31:01.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:31:01.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:31:01.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:31:01.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:31:01.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:31:01.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:31:01.463 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:31:01.463 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:31:01.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:31:01.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:31:01.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:31:01.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-08-19T14:31:01.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-08-19T14:31:01.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-08-19T14:31:01.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:31:01.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:31:01.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-08-19T14:31:01.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-08-19T14:31:01.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672970' 2024-08-19T14:31:01.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:31:01.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:31:01.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-08-19T14:31:01.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-08-19T14:31:01.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672970 2-64424509447' 2024-08-19T14:31:01.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:31:01.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-08-19T14:31:01.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:31:01.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:31:01.818 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-08-19T14:31:01.818 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:31:01.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-08-19T14:31:01.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-08-19T14:31:01.820 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836499 2024-08-19T14:31:01.820 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:31:02.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-08-19T14:31:02.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:31:03.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:31:03.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:31:03.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-08-19T14:31:03.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:31:03.356 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-08-19T14:31:03.356 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:31:03.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:31:03.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-08-19T14:31:03.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:31:03.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-08-19T14:31:03.359 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672970 2024-08-19T14:31:03.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-08-19T14:31:03.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:31:03.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-08-19T14:31:03.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:31:03.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-08-19T14:31:03.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:31:03.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:31:03.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-08-19T14:31:03.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:31:03.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-08-19T14:31:03.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-08-19T14:31:03.629 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509447 2024-08-19T14:31:03.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:31:03.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-08-19T14:31:03.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:31:03.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:31:03.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:31:04.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:31:04.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:31:04.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:31:04.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:31:04.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:31:04.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:31:04.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:31:04.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:31:04.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-19T14:31:04.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:31:04.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:31:04.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:31:04.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-19T14:31:04.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-19T14:31:04.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:31:04.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-19T14:31:04.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:31:04.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:31:04.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:31:04.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:31:04.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:31:04.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:31:04.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:31:04.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:31:04.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:31:04.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:31:05.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-19T14:31:05.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:31:05.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:31:05.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:31:05.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-19T14:31:05.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-19T14:31:05.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T14:31:05.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T14:31:05.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T14:31:05.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T14:31:05.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T14:31:05.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T14:31:05.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T14:31:05.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T14:31:05.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T14:31:05.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-19T14:31:05.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:31:05.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-19T14:31:06.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:31:06.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:31:06.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:31:06.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:31:06.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:31:06.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:31:06.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:31:06.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:31:06.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-19T14:31:06.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:31:06.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:31:06.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:31:06.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-19T14:31:06.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-19T14:31:06.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T14:31:06.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T14:31:06.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T14:31:06.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T14:31:06.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T14:31:06.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T14:31:06.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T14:31:07.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T14:31:07.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T14:31:07.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-08-19T14:31:07.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:31:07.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-19T14:31:07.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:31:07.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:31:07.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:31:07.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:31:07.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:31:07.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:31:07.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:31:07.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:31:07.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:31:07.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:31:07.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:31:07.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:31:08.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:31:08.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:31:08.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:31:08.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:502: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2024-08-19T14:31:08.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:503: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2024-08-19T14:31:08.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:504: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-08-19T14:31:08.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-08-19T14:31:08.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-08-19T14:31:08.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-08-19T14:31:08.183 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-08-19T14:31:08.184 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:31:08.184 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:31:08.184 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:31:08.184 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:31:08.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T14:31:08.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:31:08.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T14:31:08.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T14:31:08.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:31:08.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:31:08.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T14:31:08.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:31:08.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T14:31:08.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:31:08.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:31:08.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T14:31:08.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:31:08.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T14:31:08.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:31:08.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:31:08.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:31:08.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:31:08.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:31:09.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:31:09.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T14:31:09.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:31:09.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:31:09.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T14:31:09.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:31:09.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:31:09.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-08-19T14:31:10.233 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-08-19T14:31:10.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-08-19T14:31:10.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:31:10.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:31:10.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T14:31:10.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:31:10.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:31:10.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:31:10.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:31:10.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:31:10.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:31:10.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:31:10.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:31:10.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:31:10.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:31:10.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:31:10.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:31:10.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:31:10.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:31:10.757 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:31:10.757 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:31:10.758 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:31:10.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:31:10.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:31:10.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:31:10.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:31:10.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:31:10.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:31:10.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:31:10.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:31:10.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:31:10.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:31:10.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-08-19T14:31:10.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:31:10.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T14:31:10.760 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T14:31:10.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-08-19T14:31:10.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T14:31:10.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T14:31:10.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:31:10.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:31:10.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:31:10.783 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:10.782+0000 7fc8469fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:31:10.786 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:10.784+0000 7fc8469fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:31:10.787 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:10.786+0000 7fc8469fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:31:11.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T14:31:11.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:31:11.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:31:11.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:31:11.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:31:11.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:31:11.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:31:11.028 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:31:11.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:31:11.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:31:11.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:31:11.314 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:11.310+0000 7fc8469fc640 -1 Falling back to public interface 2024-08-19T14:31:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:31:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:31:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:31:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:31:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:31:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:31:12.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:31:13.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:31:13.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:31:13.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:31:13.581 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:31:13.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:31:13.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:31:13.639 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:13.638+0000 7fc8469fc640 -1 osd.0 20 log_to_monitors true 2024-08-19T14:31:13.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:31:14.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:31:14.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:31:14.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:31:14.912 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:31:14.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:31:14.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:31:15.184 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/1781854814,v1:127.0.0.1:6803/1781854814] [v2:127.0.0.1:6804/1781854814,v1:127.0.0.1:6805/1781854814] exists,up 4547bd6b-d083-4d49-a062-1439bb29fe29 2024-08-19T14:31:15.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:31:15.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:31:15.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:31:15.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:31:15.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:31:15.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:31:15.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:31:15.185 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:31:15.185 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:31:15.185 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:31:15.185 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:31:15.185 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:31:15.186 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:31:15.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:31:15.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:31:15.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:31:15.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:31:15.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:31:15.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:31:15.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:31:15.538 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:31:15.538 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:31:15.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:31:15.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:31:15.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:31:15.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215106 2024-08-19T14:31:15.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215106 2024-08-19T14:31:15.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106' 2024-08-19T14:31:15.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:31:15.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:31:15.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-08-19T14:31:15.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-08-19T14:31:15.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672974' 2024-08-19T14:31:15.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:31:15.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:31:15.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-08-19T14:31:15.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-08-19T14:31:15.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672974 2-64424509451' 2024-08-19T14:31:15.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:31:15.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215106 2024-08-19T14:31:15.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:31:15.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:31:15.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215106 2024-08-19T14:31:15.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:31:15.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215106 2024-08-19T14:31:15.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-08-19T14:31:15.849 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 103079215106 2024-08-19T14:31:15.849 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:31:16.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 103079215106 2024-08-19T14:31:16.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:31:17.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:31:17.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:31:17.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215106 -lt 103079215106 2024-08-19T14:31:17.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:31:17.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-08-19T14:31:17.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:31:17.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:31:17.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-08-19T14:31:17.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:31:17.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-08-19T14:31:17.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-08-19T14:31:17.393 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949672974 2024-08-19T14:31:17.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:31:17.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-08-19T14:31:17.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:31:17.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-08-19T14:31:17.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:31:17.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:31:17.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-08-19T14:31:17.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:31:17.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-08-19T14:31:17.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-08-19T14:31:17.661 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509451 2024-08-19T14:31:17.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:31:17.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-08-19T14:31:17.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:31:17.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:31:17.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:31:18.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:31:18.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:31:18.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:31:18.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:31:18.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:31:18.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:31:18.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:31:18.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:31:18.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:31:18.522 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:31:18.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:31:18.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:31:18.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:31:18.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:31:18.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:31:18.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2024-08-19T14:31:18.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-08-19T14:31:18.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T14:31:18.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-08-19T14:31:18.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:31:19.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2024-08-19T14:31:19.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2024-08-19T14:31:19.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:31:19.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:31:19.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:31:19.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:31:19.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: local primary=1 2024-08-19T14:31:19.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2024-08-19T14:31:19.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:31:19.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:31:19.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:31:19.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:31:19.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2024-08-19T14:30:59.535901+0000 2024-08-19T14:31:19.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 schedule-scrub 2024-08-19T14:31:19.782 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T14:31:19.782 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": false, 2024-08-19T14:31:19.782 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T14:31:19.783 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-12T14:29:39.783119+0000" 2024-08-19T14:31:19.783 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T14:31:19.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2024-08-19T14:30:59.535901+0000 2024-08-19T14:31:19.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-08-19T14:31:19.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:30:59.535901+0000 2024-08-19T14:31:19.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:31:19.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:31:19.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:31:19.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:31:19.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:31:19.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:31:19.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:31:19.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:31:20.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:30:59.535901+0000 '>' 2024-08-19T14:30:59.535901+0000 2024-08-19T14:31:20.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:31:21.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:31:21.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:31:21.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:31:21.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:31:21.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:31:21.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:31:21.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:31:21.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:30:59.535901+0000 '>' 2024-08-19T14:30:59.535901+0000 2024-08-19T14:31:21.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:31:22.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:31:22.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:31:22.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T14:31:22.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:31:22.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:31:22.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:31:22.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:31:22.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:31:19.912725+0000 '>' 2024-08-19T14:30:59.535901+0000 2024-08-19T14:31:22.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:31:22.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-08-19T14:31:22.852 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T14:31:22.852 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean+inconsistent 2024-08-19T14:31:19.912798+0000 20'1 25:51 [1,0] 1 [1,0] 1 20'1 2024-08-19T14:31:19.912725+0000 0'0 2024-08-19T14:30:59.535901+0000 0 0 queued for deep scrub 1 0 2024-08-19T14:31:22.852 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:31:22.852 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T14:31:22.853 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:31:22.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_scrub: sleep 2 2024-08-19T14:31:24.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-08-19T14:31:25.126 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T14:31:25.127 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean+inconsistent 2024-08-19T14:31:19.912798+0000 20'1 25:51 [1,0] 1 [1,0] 1 20'1 2024-08-19T14:31:19.912725+0000 0'0 2024-08-19T14:30:59.535901+0000 0 0 queued for deep scrub 1 0 2024-08-19T14:31:25.127 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:31:25.127 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T14:31:25.127 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:31:25.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:521: TEST_auto_repair_bluestore_scrub: sleep 2 2024-08-19T14:31:27.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-08-19T14:31:27.396 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T14:31:27.396 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-08-19T14:31:21.081591+0000 20'1 25:66 [1,0] 1 [1,0] 1 20'1 2024-08-19T14:31:20.912583+0000 20'1 2024-08-19T14:31:20.912583+0000 0 1 periodic scrub scheduled @ 2024-08-20T14:31:20.912583+0000 1 0 2024-08-19T14:31:27.397 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:31:27.397 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T14:31:27.397 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:31:27.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_scrub: sleep 5 2024-08-19T14:31:32.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-08-19T14:31:32.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:31:32.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:31:32.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:31:32.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:31:32.411 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:31:32.411 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:31:32.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:31:32.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:31:32.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:31:32.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:31:32.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:31:32.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:31:32.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:31:32.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:31:32.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:31:32.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:31:32.777 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:31:32.777 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:31:32.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:31:32.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:31:32.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:31:32.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215110 2024-08-19T14:31:32.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215110 2024-08-19T14:31:32.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215110' 2024-08-19T14:31:32.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:31:32.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:31:32.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-08-19T14:31:32.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-08-19T14:31:32.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215110 1-42949672978' 2024-08-19T14:31:32.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:31:32.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:31:33.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-08-19T14:31:33.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-08-19T14:31:33.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215110 1-42949672978 2-64424509455' 2024-08-19T14:31:33.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:31:33.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215110 2024-08-19T14:31:33.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:31:33.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:31:33.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215110 2024-08-19T14:31:33.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:31:33.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215110 2024-08-19T14:31:33.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215110' 2024-08-19T14:31:33.100 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 103079215110 2024-08-19T14:31:33.100 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:31:33.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215109 -lt 103079215110 2024-08-19T14:31:33.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:31:34.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:31:34.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:31:34.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215111 -lt 103079215110 2024-08-19T14:31:34.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:31:34.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-08-19T14:31:34.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:31:34.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:31:34.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-08-19T14:31:34.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:31:34.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-08-19T14:31:34.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-08-19T14:31:34.673 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672978 2024-08-19T14:31:34.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:31:34.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-08-19T14:31:34.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:31:34.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-08-19T14:31:34.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:31:34.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:31:34.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-08-19T14:31:34.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:31:34.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-08-19T14:31:34.944 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509455 2024-08-19T14:31:34.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-08-19T14:31:34.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:31:35.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509455 2024-08-19T14:31:35.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:31:35.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:31:35.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:31:35.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:31:35.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:31:35.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:31:35.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:31:35.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:31:35.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:31:35.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:31:35.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:31:35.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:31:35.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:31:35.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:31:35.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:31:36.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:31:36.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:31:36.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:31:36.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:525: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-08-19T14:31:36.391 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T14:31:36.391 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-08-19T14:31:21.081591+0000 20'1 25:66 [1,0] 1 [1,0] 1 20'1 2024-08-19T14:31:20.912583+0000 20'1 2024-08-19T14:31:20.912583+0000 0 1 periodic scrub scheduled @ 2024-08-20T14:31:20.912583+0000 1 0 2024-08-19T14:31:36.391 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:31:36.392 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T14:31:36.392 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:31:36.403 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-08-19T14:31:36.403 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-08-19T14:31:36.403 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-08-19T14:31:36.403 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-08-19T14:31:36.403 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:31:36.403 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:31:36.403 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:31:36.403 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:31:36.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T14:31:36.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:31:36.698 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T14:31:36.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-08-19T14:31:36.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:31:36.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:31:36.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T14:31:36.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:31:36.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-08-19T14:31:36.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:31:36.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:31:36.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T14:31:36.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:31:36.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T14:31:36.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:31:36.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:31:36.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:31:36.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:31:36.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:31:37.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:31:37.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-08-19T14:31:37.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:31:37.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:31:37.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T14:31:37.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:31:37.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:31:37.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-08-19T14:31:38.234 INFO:tasks.workunit.client.0.smithi102.stdout:_ 2024-08-19T14:31:38.234 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 2024-08-19T14:31:38.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-08-19T14:31:38.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:31:38.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:31:38.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T14:31:38.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:31:38.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:31:38.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:31:38.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:31:38.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:31:38.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:31:38.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:31:38.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:31:38.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:31:38.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:31:38.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:31:38.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:31:38.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:31:38.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:31:38.503 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:31:38.503 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:31:38.503 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:31:38.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:31:38.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:31:38.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:31:38.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:31:38.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:31:38.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:31:38.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:31:38.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:31:38.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:31:38.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:31:38.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-08-19T14:31:38.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:31:38.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T14:31:38.504 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T14:31:38.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-08-19T14:31:38.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T14:31:38.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T14:31:38.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:31:38.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:31:38.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:31:38.535 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:38.534+0000 7fb0c301a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:31:38.537 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:38.537+0000 7fb0c301a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:31:38.539 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:38.538+0000 7fb0c301a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:31:38.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T14:31:38.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:31:38.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:31:38.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:31:38.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:31:38.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:31:38.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:31:38.786 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:31:38.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:31:38.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:31:39.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:31:39.071 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:39.071+0000 7fb0c301a640 -1 Falling back to public interface 2024-08-19T14:31:40.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:31:40.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:31:40.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:31:40.053 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:31:40.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:31:40.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:31:40.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:31:41.300 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:41.300+0000 7fb0c301a640 -1 osd.0 25 log_to_monitors true 2024-08-19T14:31:41.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:31:41.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:31:41.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:31:41.327 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:31:41.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:31:41.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:31:41.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:31:42.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:31:42.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:31:42.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:31:42.614 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:31:42.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:31:42.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:31:42.925 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6802/3320062838,v1:127.0.0.1:6803/3320062838] [v2:127.0.0.1:6804/3320062838,v1:127.0.0.1:6805/3320062838] exists,up 4547bd6b-d083-4d49-a062-1439bb29fe29 2024-08-19T14:31:42.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:31:42.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:31:42.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:31:42.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:31:42.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:31:42.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:31:42.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:31:42.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:31:42.926 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:31:42.926 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:31:42.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:31:42.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:31:42.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:31:43.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:31:43.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:31:43.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:31:43.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:31:43.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:31:43.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:31:43.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:31:43.280 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:31:43.281 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:31:43.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:31:43.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:31:43.281 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:31:43.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-08-19T14:31:43.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-08-19T14:31:43.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586' 2024-08-19T14:31:43.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:31:43.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:31:43.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-08-19T14:31:43.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-08-19T14:31:43.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-42949672981' 2024-08-19T14:31:43.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:31:43.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:31:43.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-08-19T14:31:43.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-08-19T14:31:43.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-42949672981 2-64424509458' 2024-08-19T14:31:43.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:31:43.591 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051586 2024-08-19T14:31:43.591 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:31:43.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:31:43.592 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051586 2024-08-19T14:31:43.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:31:43.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-08-19T14:31:43.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-08-19T14:31:43.594 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 124554051586 2024-08-19T14:31:43.594 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:31:43.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-08-19T14:31:43.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:31:44.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:31:44.862 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:31:45.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-08-19T14:31:45.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:31:45.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-08-19T14:31:45.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:31:45.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:31:45.136 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-08-19T14:31:45.136 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:31:45.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-08-19T14:31:45.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-08-19T14:31:45.137 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949672981 2024-08-19T14:31:45.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:31:45.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-08-19T14:31:45.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:31:45.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-08-19T14:31:45.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:31:45.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:31:45.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-08-19T14:31:45.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:31:45.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-08-19T14:31:45.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-08-19T14:31:45.444 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509458 2024-08-19T14:31:45.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:31:45.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509458 2024-08-19T14:31:45.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:31:45.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:31:45.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:31:46.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:31:46.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:31:46.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:31:46.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:31:46.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:31:46.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:31:46.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:31:46.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:31:46.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:31:46.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:31:46.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:31:46.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:31:46.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:31:46.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:31:46.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:31:46.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T14:31:46.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T14:31:46.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-08-19T14:31:46.713 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:31:19.911+0000 7f2c22a8d640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] planned DEEP_SCRUB_ON_ERROR] scrubber: scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2024-08-19T14:31:46.713 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:31:19.911+0000 7f2c22a8d640 15 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish Try to auto repair after scrub errors 2024-08-19T14:31:46.713 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:31:19.911+0000 7f2c22a8d640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-08-19T14:31:46.713 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:31:20.911+0000 7f2c22a8d640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-08-19T14:31:46.714 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:31:20.911+0000 7f2c22a8d640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-08-19T14:31:46.714 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:31:20.911+0000 7f2c22a8d640 15 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-08-19T14:31:46.714 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:31:20.911+0000 7f2c22a8d640 20 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-08-19T14:31:46.714 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:31:20.911+0000 7f2c22a8d640 19 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-08-19T14:31:46.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2024-08-19T14:31:46.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2024-08-19T14:31:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: COUNT=1 2024-08-19T14:31:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_scrub: test 1 = 1 2024-08-19T14:31:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T14:31:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:31:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:31:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:31:46.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:31:46.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:31:46.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:31:46.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:31:46.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:31:46.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:31:46.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:31:46.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:31:46.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:31:46.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:31:46.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:31:46.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:31:46.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:31:46.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:31:46.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:31:46.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:31:46.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:31:46.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:31:46.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:31:46.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:31:46.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:31:46.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:31:46.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:31:46.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:31:46.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:31:46.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T14:31:46.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T14:31:46.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T14:31:46.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T14:31:46.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:31:46.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:31:46.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:31:46.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:31:46.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:31:46.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:31:46.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:31:46.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:31:46.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:31:46.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:31:46.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:31:46.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:31:46.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:31:46.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:31:46.979 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:31:46.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:31:46.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:31:46.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:31:46.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:31:46.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:31:46.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:31:46.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:31:46.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:31:46.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:31:46.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:31:46.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:31:46.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:31:46.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:31:46.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T14:31:46.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T14:31:46.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:31:46.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:31:46.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T14:31:46.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T14:31:46.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T14:31:46.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T14:31:46.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T14:31:46.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2024-08-19T14:31:46.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2024-08-19T14:31:46.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: TEST_auto_repair_bluestore_tag: local poolname=testpool 2024-08-19T14:31:46.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2024-08-19T14:31:46.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T14:31:46.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T14:31:46.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T14:31:46.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T14:31:46.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T14:31:46.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-08-19T14:31:47.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T14:31:47.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:31:47.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:31:47.300 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:31:47.300 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:31:47.300 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:31:47.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:31:47.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T14:31:47.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T14:31:47.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T14:31:47.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:31:47.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:31:47.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T14:31:47.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T14:31:47.473 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:31:47.473 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:31:47.474 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:31:47.474 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:31:47.474 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:31:47.475 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:31:47.475 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:31:47.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:31:47.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T14:31:47.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T14:31:47.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:31:47.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:31:47.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T14:31:47.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T14:31:47.554 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:31:47.554 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:31:47.554 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:31:47.554 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:31:47.554 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:31:47.554 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:31:47.555 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:31:47.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:31:47.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T14:31:47.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:388: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2024-08-19T14:31:47.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T14:31:47.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T14:31:47.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T14:31:47.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T14:31:47.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T14:31:47.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T14:31:48.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T14:31:48.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:31:48.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:31:48.441 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:31:48.441 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:31:48.441 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:31:48.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:31:48.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:31:48.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:31:48.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:395: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-08-19T14:31:48.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: seq 0 2 2024-08-19T14:31:48.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-08-19T14:31:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-08-19T14:31:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:31:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:31:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T14:31:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:31:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:31:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:31:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:31:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:31:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:31:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:31:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:31:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:31:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:31:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:31:48.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:31:48.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:31:48.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:31:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:31:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:31:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:31:48.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:31:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:31:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:31:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:31:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:31:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:31:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:31:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:31:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:31:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:31:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:31:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:31:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-08-19T14:31:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:31:48.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:31:48.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1f2360a0-17a2-47e8-83b2-fa195e397bf2 2024-08-19T14:31:48.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1f2360a0-17a2-47e8-83b2-fa195e397bf2' 2024-08-19T14:31:48.477 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 1f2360a0-17a2-47e8-83b2-fa195e397bf2 2024-08-19T14:31:48.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:31:48.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBUV8Nm6sVcHRAA01lshf5IvXHhk6wUqPurQQ== 2024-08-19T14:31:48.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBUV8Nm6sVcHRAA01lshf5IvXHhk6wUqPurQQ=="}' 2024-08-19T14:31:48.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1f2360a0-17a2-47e8-83b2-fa195e397bf2 -i td/osd-scrub-repair/0/new.json 2024-08-19T14:31:48.689 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:31:48.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T14:31:48.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBUV8Nm6sVcHRAA01lshf5IvXHhk6wUqPurQQ== --osd-uuid 1f2360a0-17a2-47e8-83b2-fa195e397bf2 2024-08-19T14:31:48.730 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:48.730+0000 7f928cf50640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:31:48.739 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:48.739+0000 7f928cf50640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:31:48.740 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:48.740+0000 7f928cf50640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:31:48.741 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:48.741+0000 7f928cf50640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:31:48.741 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:48.741+0000 7f928cf50640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:31:48.741 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:48.741+0000 7f928cf50640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:31:48.832 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:48.831+0000 7f928cf50640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T14:31:54.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T14:31:54.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:31:54.181 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T14:31:54.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T14:31:54.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:31:54.526 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T14:31:54.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T14:31:54.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-08-19T14:31:54.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:31:54.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:31:54.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:31:54.555 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:54.555+0000 7faa3560f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:31:54.581 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:54.581+0000 7faa3560f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:31:54.582 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:54.582+0000 7faa3560f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:31:54.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T14:31:54.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:31:54.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:31:54.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:31:54.792 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:31:54.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:31:54.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:31:54.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:31:54.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:31:54.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:31:55.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:31:55.101 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:55.101+0000 7faa3560f640 -1 Falling back to public interface 2024-08-19T14:31:56.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:31:56.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:31:56.094 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:31:56.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:31:56.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:31:56.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:31:56.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:31:57.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:31:57.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:31:57.369 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:31:57.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:31:57.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:31:57.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:31:57.415 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:57.414+0000 7faa3560f640 -1 osd.0 0 log_to_monitors true 2024-08-19T14:31:57.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:31:58.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:31:58.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:31:58.690 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:31:58.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:31:58.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:31:58.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:31:58.974 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1670236621,v1:127.0.0.1:6803/1670236621] [v2:127.0.0.1:6804/1670236621,v1:127.0.0.1:6805/1670236621] exists,up 1f2360a0-17a2-47e8-83b2-fa195e397bf2 2024-08-19T14:31:58.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:31:58.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:31:58.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:31:58.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-08-19T14:31:58.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-08-19T14:31:58.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:31:58.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:31:58.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T14:31:58.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:31:58.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:31:58.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:31:58.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:31:58.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:31:58.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:31:58.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:31:58.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:31:58.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:31:58.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:31:58.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:31:58.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:31:58.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:31:58.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:31:58.980 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:31:58.980 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:31:58.980 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:31:58.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:31:58.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:31:58.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:31:58.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:31:58.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:31:58.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:31:58.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:31:58.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:31:58.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:31:58.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:31:58.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:31:58.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:31:58.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-08-19T14:31:58.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:31:58.982 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:31:58.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=06edcd68-42e8-4705-8339-ae1ebb00118b 2024-08-19T14:31:58.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 06edcd68-42e8-4705-8339-ae1ebb00118b' 2024-08-19T14:31:58.982 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 06edcd68-42e8-4705-8339-ae1ebb00118b 2024-08-19T14:31:58.982 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:31:58.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBeV8Nm0Tg7OxAAHBRyd7oo1i4hQWGXkTdPjw== 2024-08-19T14:31:58.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBeV8Nm0Tg7OxAAHBRyd7oo1i4hQWGXkTdPjw=="}' 2024-08-19T14:31:58.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 06edcd68-42e8-4705-8339-ae1ebb00118b -i td/osd-scrub-repair/1/new.json 2024-08-19T14:31:59.359 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:31:59.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T14:31:59.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBeV8Nm0Tg7OxAAHBRyd7oo1i4hQWGXkTdPjw== --osd-uuid 06edcd68-42e8-4705-8339-ae1ebb00118b 2024-08-19T14:31:59.402 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:59.400+0000 7f44a6e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:31:59.434 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:59.433+0000 7f44a6e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:31:59.435 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:59.434+0000 7f44a6e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:31:59.436 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:59.434+0000 7f44a6e1d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:31:59.436 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:59.435+0000 7f44a6e1d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:31:59.436 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:59.435+0000 7f44a6e1d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:31:59.501 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:31:59.500+0000 7f44a6e1d640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T14:32:04.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T14:32:04.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:32:04.957 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T14:32:04.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T14:32:04.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:32:05.296 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T14:32:05.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T14:32:05.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-08-19T14:32:05.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:32:05.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:32:05.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:32:05.325 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:05.325+0000 7f7ca11a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:32:05.327 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:05.326+0000 7f7ca11a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:32:05.328 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:05.328+0000 7f7ca11a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:32:05.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T14:32:05.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:32:05.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:32:05.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:32:05.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:32:05.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:32:05.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:32:05.567 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:32:05.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:32:05.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:32:05.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:32:05.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:05.861+0000 7f7ca11a3640 -1 Falling back to public interface 2024-08-19T14:32:06.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:32:06.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:32:06.833 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:32:06.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:32:06.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:32:06.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:32:07.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:32:08.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:32:08.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:32:08.111 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:32:08.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:32:08.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:32:08.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:32:08.225 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:08.224+0000 7f7ca11a3640 -1 osd.1 0 log_to_monitors true 2024-08-19T14:32:08.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:32:09.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:32:09.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:32:09.418 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:32:09.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:32:09.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:32:09.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:32:09.692 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/599382111,v1:127.0.0.1:6811/599382111] [v2:127.0.0.1:6812/599382111,v1:127.0.0.1:6813/599382111] exists,up 06edcd68-42e8-4705-8339-ae1ebb00118b 2024-08-19T14:32:09.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:32:09.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:32:09.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:32:09.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-08-19T14:32:09.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-08-19T14:32:09.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:32:09.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:32:09.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T14:32:09.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:32:09.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T14:32:09.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:32:09.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:32:09.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:32:09.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:32:09.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T14:32:09.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T14:32:09.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:32:09.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:32:09.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:32:09.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:32:09.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:32:09.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:32:09.699 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:32:09.700 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:32:09.700 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:32:09.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:32:09.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:32:09.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:32:09.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:32:09.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:32:09.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:32:09.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:32:09.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:32:09.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:32:09.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:32:09.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:32:09.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:32:09.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-08-19T14:32:09.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T14:32:09.701 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:32:09.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=93e98a73-cec0-45df-bcd0-e3099601b51a 2024-08-19T14:32:09.701 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 93e98a73-cec0-45df-bcd0-e3099601b51a 2024-08-19T14:32:09.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 93e98a73-cec0-45df-bcd0-e3099601b51a' 2024-08-19T14:32:09.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:32:09.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBpV8NmABCNKhAA4qR+/KZ1C1xhes0GxnkBfg== 2024-08-19T14:32:09.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBpV8NmABCNKhAA4qR+/KZ1C1xhes0GxnkBfg=="}' 2024-08-19T14:32:09.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 93e98a73-cec0-45df-bcd0-e3099601b51a -i td/osd-scrub-repair/2/new.json 2024-08-19T14:32:10.053 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:32:10.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-08-19T14:32:10.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBpV8NmABCNKhAA4qR+/KZ1C1xhes0GxnkBfg== --osd-uuid 93e98a73-cec0-45df-bcd0-e3099601b51a 2024-08-19T14:32:10.097 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:10.096+0000 7fae1420f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:32:10.131 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:10.131+0000 7fae1420f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:32:10.132 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:10.132+0000 7fae1420f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:32:10.133 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:10.132+0000 7fae1420f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:32:10.133 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:10.132+0000 7fae1420f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:32:10.133 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:10.132+0000 7fae1420f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:32:10.187 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:10.187+0000 7fae1420f640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-08-19T14:32:15.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-08-19T14:32:15.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:32:15.695 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T14:32:15.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T14:32:15.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:32:16.037 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T14:32:16.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T14:32:16.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-08-19T14:32:16.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:32:16.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:32:16.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:32:16.066 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:16.066+0000 7fb690e30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:32:16.068 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:16.068+0000 7fb690e30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:32:16.070 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:16.069+0000 7fb690e30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:32:16.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T14:32:16.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:32:16.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T14:32:16.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:32:16.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:32:16.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:32:16.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:32:16.311 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:32:16.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:32:16.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:32:16.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:32:16.597 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:16.597+0000 7fb690e30640 -1 Falling back to public interface 2024-08-19T14:32:17.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:32:17.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:32:17.592 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:32:17.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:32:17.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:32:17.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:32:17.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:32:18.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:32:18.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:32:18.867 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:32:18.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:32:18.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:32:18.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:32:18.994 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:18.993+0000 7fb690e30640 -1 osd.2 0 log_to_monitors true 2024-08-19T14:32:19.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:32:20.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:32:20.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:32:20.220 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:32:20.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:32:20.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:32:20.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:32:20.525 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2572230480,v1:127.0.0.1:6819/2572230480] [v2:127.0.0.1:6820/2572230480,v1:127.0.0.1:6821/2572230480] exists,up 93e98a73-cec0-45df-bcd0-e3099601b51a 2024-08-19T14:32:20.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:32:20.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:32:20.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:32:20.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2024-08-19T14:32:20.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-08-19T14:32:20.923 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'testpool' created 2024-08-19T14:32:20.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T14:32:21.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2024-08-19T14:32:22.355 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 2 2024-08-19T14:32:22.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:402: TEST_auto_repair_bluestore_tag: wait_for_clean 2024-08-19T14:32:22.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:32:22.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:32:22.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:32:22.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:32:22.367 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:32:22.367 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:32:22.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:32:22.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:32:22.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:32:22.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:32:22.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:32:22.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:32:22.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:32:22.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:32:22.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:32:22.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:32:22.851 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:32:22.851 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:32:22.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:32:22.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:32:22.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:32:23.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836486 2024-08-19T14:32:23.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836486 2024-08-19T14:32:23.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486' 2024-08-19T14:32:23.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:32:23.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:32:23.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-08-19T14:32:23.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-08-19T14:32:23.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486 1-42949672964' 2024-08-19T14:32:23.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:32:23.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:32:23.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509442 2024-08-19T14:32:23.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509442 2024-08-19T14:32:23.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486 1-42949672964 2-64424509442' 2024-08-19T14:32:23.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:32:23.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836486 2024-08-19T14:32:23.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:32:23.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:32:23.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836486 2024-08-19T14:32:23.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:32:23.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836486 2024-08-19T14:32:23.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-08-19T14:32:23.250 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836486 2024-08-19T14:32:23.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:32:23.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836486 2024-08-19T14:32:23.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:32:24.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:32:24.604 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:32:24.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836486 2024-08-19T14:32:24.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:32:24.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-08-19T14:32:24.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:32:24.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:32:24.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-08-19T14:32:24.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:32:24.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-08-19T14:32:24.883 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672964 2024-08-19T14:32:24.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-08-19T14:32:24.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:32:25.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672964 2024-08-19T14:32:25.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:32:25.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509442 2024-08-19T14:32:25.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:32:25.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:32:25.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509442 2024-08-19T14:32:25.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:32:25.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509442 2024-08-19T14:32:25.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-08-19T14:32:25.150 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509442 2024-08-19T14:32:25.151 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:32:25.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509442 -lt 64424509442 2024-08-19T14:32:25.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:32:25.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:32:25.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:32:25.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:32:25.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:32:25.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:32:25.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:32:25.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:32:25.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:32:25.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:32:25.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:32:26.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-19T14:32:26.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:32:26.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:32:26.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:32:26.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-19T14:32:26.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-19T14:32:26.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:32:26.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-19T14:32:26.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:32:26.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:32:26.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:32:26.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:32:26.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:32:26.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:32:26.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:32:26.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:32:26.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:32:26.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:32:26.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:32:26.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:32:26.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:32:26.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:32:27.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:32:27.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:32:27.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:32:27.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:405: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2024-08-19T14:32:27.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: TEST_auto_repair_bluestore_tag: echo ABCDEF 2024-08-19T14:32:27.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:407: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-08-19T14:32:27.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-08-19T14:32:27.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-08-19T14:32:27.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-08-19T14:32:27.155 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-08-19T14:32:27.155 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:32:27.155 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:32:27.155 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:32:27.155 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:32:27.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T14:32:27.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:32:27.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T14:32:27.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T14:32:27.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:32:27.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:32:27.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T14:32:27.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:32:27.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T14:32:27.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:32:27.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:32:27.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T14:32:27.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:32:27.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T14:32:27.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:32:27.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:32:27.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:32:27.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:32:27.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:32:28.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:32:28.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T14:32:28.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:32:28.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:32:28.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T14:32:28.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:32:28.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:32:28.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-08-19T14:32:29.347 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-08-19T14:32:29.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-08-19T14:32:29.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:32:29.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:32:29.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T14:32:29.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:32:29.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:32:29.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:32:29.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:32:29.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:32:29.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:32:29.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:32:29.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:32:29.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:32:29.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:32:29.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:32:29.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:32:29.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:32:29.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:32:29.870 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:32:29.870 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:32:29.870 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:32:29.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:32:29.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:32:29.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:32:29.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:32:29.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:32:29.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:32:29.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:32:29.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:32:29.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:32:29.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:32:29.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-08-19T14:32:29.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:32:29.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T14:32:29.872 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T14:32:29.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-08-19T14:32:29.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T14:32:29.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T14:32:29.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:32:29.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:32:29.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:32:29.902 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:29.902+0000 7fd9e2c9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:32:29.905 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:29.904+0000 7fd9e2c9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:32:29.906 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:29.906+0000 7fd9e2c9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:32:30.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T14:32:30.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:32:30.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:32:30.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:32:30.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:32:30.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:32:30.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:32:30.152 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:32:30.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:32:30.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:32:30.425 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:30.425+0000 7fd9e2c9c640 -1 Falling back to public interface 2024-08-19T14:32:30.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:32:31.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:32:31.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:32:31.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:32:31.437 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:32:31.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:32:31.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:32:31.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:32:32.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:32:32.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:32:32.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:32:32.711 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:32:32.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:32:32.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:32:32.845 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:32.844+0000 7fd9e2c9c640 -1 osd.0 21 log_to_monitors true 2024-08-19T14:32:33.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:32:34.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:32:34.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:32:34.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:32:34.051 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:32:34.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:32:34.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:32:34.334 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/639564583,v1:127.0.0.1:6803/639564583] [v2:127.0.0.1:6804/639564583,v1:127.0.0.1:6805/639564583] exists,up 1f2360a0-17a2-47e8-83b2-fa195e397bf2 2024-08-19T14:32:34.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:32:34.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:32:34.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:32:34.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:32:34.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:32:34.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:32:34.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:32:34.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:32:34.335 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:32:34.335 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:32:34.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:32:34.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:32:34.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:32:34.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:32:34.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:32:34.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:32:34.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:32:34.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:32:34.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:32:34.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:32:34.689 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:32:34.689 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:32:34.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:32:34.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:32:34.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:32:34.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-08-19T14:32:34.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-08-19T14:32:34.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-08-19T14:32:34.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:32:34.816 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:32:34.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672968 2024-08-19T14:32:34.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672968 2024-08-19T14:32:34.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672968' 2024-08-19T14:32:34.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:32:34.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:32:35.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-08-19T14:32:35.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-08-19T14:32:35.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672968 2-64424509445' 2024-08-19T14:32:35.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:32:35.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-08-19T14:32:35.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:32:35.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:32:35.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-08-19T14:32:35.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:32:35.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-08-19T14:32:35.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-08-19T14:32:35.026 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 107374182402 2024-08-19T14:32:35.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:32:35.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-08-19T14:32:35.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:32:36.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:32:36.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:32:36.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-08-19T14:32:36.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:32:36.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672968 2024-08-19T14:32:36.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:32:36.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:32:36.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672968 2024-08-19T14:32:36.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:32:36.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672968 2024-08-19T14:32:36.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-08-19T14:32:36.589 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949672968 2024-08-19T14:32:36.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:32:36.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672968 2024-08-19T14:32:36.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:32:36.861 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-08-19T14:32:36.861 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:32:36.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:32:36.862 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-08-19T14:32:36.862 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:32:36.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-08-19T14:32:36.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-08-19T14:32:36.863 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509445 2024-08-19T14:32:36.863 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:32:37.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509445 2024-08-19T14:32:37.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:32:37.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:32:37.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:32:37.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:32:37.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:32:37.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:32:37.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:32:37.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:32:37.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:32:37.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:32:37.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:32:37.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:32:37.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:32:37.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:32:37.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:32:38.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:32:38.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:32:38.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:32:38.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2024-08-19T14:32:38.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-08-19T14:32:38.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T14:32:38.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-08-19T14:32:38.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:32:38.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: local pgid=1.0 2024-08-19T14:32:38.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2024-08-19T14:32:38.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:32:38.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:32:38.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:32:38.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:32:38.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: local primary=1 2024-08-19T14:32:38.718 INFO:tasks.workunit.client.0.smithi102.stdout:Affected PG 1.0 w/ primary 1 2024-08-19T14:32:38.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2024-08-19T14:32:38.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2024-08-19T14:32:38.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:32:38.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:32:38.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:32:38.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:32:38.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2024-08-19T14:32:20.878005+0000 2024-08-19T14:32:38.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:417: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2024-08-19T14:32:38.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: initiate_and_fetch_state: local the_osd=osd.1 2024-08-19T14:32:38.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: initiate_and_fetch_state: local pgid=1.0 2024-08-19T14:32:38.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2024-08-19T14:32:38.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T14:32:38.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:32:38.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:32:38.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T14:32:39.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: local last_scrub=2024-08-19T14:32:20.878005+0000 2024-08-19T14:32:39.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:326: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2024-08-19T14:32:39.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-08-19T14:32:39.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-08-19T14:32:39.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-08-19T14:32:39.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=3.0 2024-08-19T14:32:39.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-08-19T14:32:39.247 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-08-19T14:32:39.247 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T14:32:39.247 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T14:32:39.247 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:32:39.247 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:32:39.247 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:32:39.247 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.1.asok 2024-08-19T14:32:39.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.57280/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2024-08-19T14:32:39.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-08-19T14:32:39.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2024-08-19T14:32:39.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-08-19T14:32:39.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-08-19T14:32:39.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_auto_repair 2024-08-19T14:32:39.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-08-19T14:32:39.333 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-08-19T14:32:39.333 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-08-19T14:32:39.333 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T14:32:39.334 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T14:32:39.334 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:32:39.334 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:32:39.334 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:32:39.334 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.1.asok 2024-08-19T14:32:39.334 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.57280/ceph-osd.1.asok config set osd_scrub_auto_repair true 2024-08-19T14:32:39.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-08-19T14:32:39.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: initiate_and_fetch_state: flush_pg_stats 2024-08-19T14:32:39.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:32:39.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:32:39.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:32:39.684 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:32:39.684 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:32:39.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:32:39.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:32:39.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:32:39.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182404 2024-08-19T14:32:39.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182404 2024-08-19T14:32:39.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404' 2024-08-19T14:32:39.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:32:39.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:32:39.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-08-19T14:32:39.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-08-19T14:32:39.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672970' 2024-08-19T14:32:39.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:32:39.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:32:40.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-08-19T14:32:40.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-08-19T14:32:40.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672970 2-64424509447' 2024-08-19T14:32:40.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:32:40.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182404 2024-08-19T14:32:40.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:32:40.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:32:40.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182404 2024-08-19T14:32:40.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:32:40.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182404 2024-08-19T14:32:40.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182404' 2024-08-19T14:32:40.006 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 107374182404 2024-08-19T14:32:40.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:32:40.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182403 -lt 107374182404 2024-08-19T14:32:40.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:32:41.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:32:41.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:32:41.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182404 2024-08-19T14:32:41.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:32:41.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-08-19T14:32:41.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:32:41.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:32:41.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-08-19T14:32:41.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:32:41.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-08-19T14:32:41.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-08-19T14:32:41.544 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672970 2024-08-19T14:32:41.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:32:41.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-08-19T14:32:41.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:32:41.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-08-19T14:32:41.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:32:41.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:32:41.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-08-19T14:32:41.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:32:41.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-08-19T14:32:41.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-08-19T14:32:41.815 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509447 2024-08-19T14:32:41.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:32:42.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-08-19T14:32:42.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: initiate_and_fetch_state: date --rfc-3339=ns 2024-08-19T14:32:42.083 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19 14:32:42.084194812+00:00 2024-08-19T14:32:42.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: get_asok_path osd.1 2024-08-19T14:32:42.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T14:32:42.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T14:32:42.084 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:32:42.085 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:32:42.085 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:32:42.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.1.asok 2024-08-19T14:32:42.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.57280/ceph-osd.1.asok schedule-deep-scrub 1.0 2024-08-19T14:32:42.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i=0 )) 2024-08-19T14:32:42.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-08-19T14:32:42.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-08-19T14:32:42.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-08-19T14:32:42.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-08-19T14:32:42.281 INFO:tasks.workunit.client.0.smithi102.stdout:{"deep":true,"must":false,"stamp":"2024-08-05T14:31:02.167232+0000"}0 ) state now: "active+clean" 2024-08-19T14:32:42.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2024-08-19T14:32:42.282 INFO:tasks.workunit.client.0.smithi102.stdout:"active+clean" 2024-08-19T14:32:42.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-08-19T14:32:42.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-08-19T14:32:42.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-08-19T14:32:42.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-08-19T14:32:42.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-08-19T14:32:42.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-08-19T14:32:42.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-08-19T14:32:42.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-08-19T14:32:42.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-08-19T14:32:42.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean"' 2024-08-19T14:32:42.689 INFO:tasks.workunit.client.0.smithi102.stdout:1 ) state now: "active+clean" 2024-08-19T14:32:42.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-08-19T14:32:42.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-08-19T14:32:42.690 INFO:tasks.workunit.client.0.smithi102.stdout:"active+clean" 2024-08-19T14:32:42.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 1 == 4 ']' 2024-08-19T14:32:42.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-08-19T14:32:42.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-08-19T14:32:42.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-08-19T14:32:42.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-08-19T14:32:42.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-08-19T14:32:43.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-08-19T14:32:43.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 2 ') state now: ' '"active+clean"' 2024-08-19T14:32:43.098 INFO:tasks.workunit.client.0.smithi102.stdout:2 ) state now: "active+clean" 2024-08-19T14:32:43.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-08-19T14:32:43.098 INFO:tasks.workunit.client.0.smithi102.stdout:"active+clean" 2024-08-19T14:32:43.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-08-19T14:32:43.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 2 == 4 ']' 2024-08-19T14:32:43.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-08-19T14:32:43.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-08-19T14:32:43.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-08-19T14:32:43.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-08-19T14:32:43.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-08-19T14:32:43.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2024-08-19T14:32:43.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 3 ') state now: ' '"active+clean+scrubbing+deep"' 2024-08-19T14:32:43.506 INFO:tasks.workunit.client.0.smithi102.stdout:3 ) state now: "active+clean+scrubbing+deep" 2024-08-19T14:32:43.506 INFO:tasks.workunit.client.0.smithi102.stdout:found scrub 2024-08-19T14:32:43.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-08-19T14:32:43.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 'found scrub' 2024-08-19T14:32:43.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: return 0 2024-08-19T14:32:43.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_tag: r=0 2024-08-19T14:32:43.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:419: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2024-08-19T14:32:43.507 INFO:tasks.workunit.client.0.smithi102.stdout:initiate_and_fetch_state ret: 0 2024-08-19T14:32:43.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2024-08-19T14:32:43.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-08-19T14:32:43.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=td/osd-scrub-repair 2024-08-19T14:32:43.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-08-19T14:32:43.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=0 2024-08-19T14:32:43.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-08-19T14:32:43.510 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.td/osd-scrub-repair 2024-08-19T14:32:43.510 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.td/osd-scrub-repair 2024-08-19T14:32:43.510 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2024-08-19T14:32:43.510 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:32:43.510 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:32:43.510 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:32:43.510 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.td/osd-scrub-repair.asok 2024-08-19T14:32:43.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.57280/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2024-08-19T14:32:43.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test == true 2024-08-19T14:32:43.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1168: test: ==: unary operator expected 2024-08-19T14:32:43.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2024-08-19T14:32:43.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2024-08-19T14:32:43.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: wait_end_of_scrub: local the_osd=osd.1 2024-08-19T14:32:43.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:361: wait_end_of_scrub: local pgid=1.0 2024-08-19T14:32:43.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i=0 )) 2024-08-19T14:32:43.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-08-19T14:32:43.573 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-08-19T14:32:43.573 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-08-19T14:32:43.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-08-19T14:32:43.676 INFO:tasks.workunit.client.0.smithi102.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-08-19T14:32:43.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-08-19T14:32:43.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-08-19T14:32:43.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 0 == 4 ']' 2024-08-19T14:32:43.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-08-19T14:32:43.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-08-19T14:32:43.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-08-19T14:32:43.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-08-19T14:32:43.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-08-19T14:32:44.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-08-19T14:32:44.083 INFO:tasks.workunit.client.0.smithi102.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-08-19T14:32:44.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-08-19T14:32:44.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-08-19T14:32:44.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 1 == 4 ']' 2024-08-19T14:32:44.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-08-19T14:32:44.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-08-19T14:32:44.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-08-19T14:32:44.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-08-19T14:32:44.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-08-19T14:32:44.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-08-19T14:32:44.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-08-19T14:32:44.492 INFO:tasks.workunit.client.0.smithi102.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-08-19T14:32:44.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-08-19T14:32:44.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 2 == 4 ']' 2024-08-19T14:32:44.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-08-19T14:32:44.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-08-19T14:32:44.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-08-19T14:32:44.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-08-19T14:32:44.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-08-19T14:32:44.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-08-19T14:32:44.899 INFO:tasks.workunit.client.0.smithi102.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-08-19T14:32:44.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-08-19T14:32:44.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-08-19T14:32:44.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 3 == 4 ']' 2024-08-19T14:32:44.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-08-19T14:32:45.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-08-19T14:32:45.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-08-19T14:32:45.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-08-19T14:32:45.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-08-19T14:32:45.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-08-19T14:32:45.308 INFO:tasks.workunit.client.0.smithi102.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-08-19T14:32:45.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-08-19T14:32:45.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-08-19T14:32:45.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 4 == 4 ']' 2024-08-19T14:32:45.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: flush_pg_stats 2024-08-19T14:32:45.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:32:45.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:32:45.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:32:45.573 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:32:45.573 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:32:45.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:32:45.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:32:45.574 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:32:45.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182406 2024-08-19T14:32:45.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182406 2024-08-19T14:32:45.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406' 2024-08-19T14:32:45.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:32:45.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:32:45.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-08-19T14:32:45.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-08-19T14:32:45.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406 1-42949672972' 2024-08-19T14:32:45.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:32:45.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:32:45.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-08-19T14:32:45.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-08-19T14:32:45.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406 1-42949672972 2-64424509450' 2024-08-19T14:32:45.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:32:45.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182406 2024-08-19T14:32:45.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:32:45.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:32:45.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182406 2024-08-19T14:32:45.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:32:45.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182406 2024-08-19T14:32:45.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182406' 2024-08-19T14:32:45.884 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 107374182406 2024-08-19T14:32:45.885 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:32:46.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182406 2024-08-19T14:32:46.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:32:47.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:32:47.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:32:47.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182406 -lt 107374182406 2024-08-19T14:32:47.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:32:47.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-08-19T14:32:47.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:32:47.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:32:47.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-08-19T14:32:47.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:32:47.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-08-19T14:32:47.426 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672972 2024-08-19T14:32:47.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-08-19T14:32:47.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:32:47.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-08-19T14:32:47.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:32:47.693 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-08-19T14:32:47.693 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:32:47.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:32:47.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-08-19T14:32:47.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:32:47.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-08-19T14:32:47.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-08-19T14:32:47.695 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509450 2024-08-19T14:32:47.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:32:47.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-08-19T14:32:47.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-08-19T14:32:48.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-08-19T14:32:48.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-08-19T14:32:48.264 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-08-19T14:32:48.264 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-08-19T14:32:48.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean"' 2024-08-19T14:32:48.371 INFO:tasks.workunit.client.0.smithi102.stdout:wait-scrub-end state now: "active+clean" 2024-08-19T14:32:48.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2024-08-19T14:32:48.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-08-19T14:32:48.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: break 2024-08-19T14:32:48.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-08-19T14:32:48.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:378: wait_end_of_scrub: return 0 2024-08-19T14:32:48.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2024-08-19T14:32:48.620 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T14:32:48.621 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean+scrubbing+deep 2024-08-19T14:32:43.197135+0000 21'1 26:44 [1,0] 1 [1,0] 1 0'0 2024-08-05T14:31:02.167232+0000 0'0 2024-08-05T14:31:02.167232+0000 0 0 deep scrubbing for 1s 0 0 2024-08-19T14:32:48.621 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T14:32:48.621 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T14:32:48.621 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T14:32:48.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-08-19T14:32:48.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-08-19T14:32:48.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-08-19T14:32:48.632 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-08-19T14:32:48.632 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:32:48.632 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:32:48.632 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:32:48.633 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:32:48.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T14:32:48.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:32:48.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T14:32:49.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-08-19T14:32:49.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:32:49.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:32:49.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T14:32:49.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:32:49.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-08-19T14:32:49.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:32:49.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:32:49.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T14:32:49.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:32:49.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T14:32:49.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:32:49.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:32:49.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:32:49.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:32:49.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:32:49.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:32:49.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-08-19T14:32:49.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:32:49.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:32:49.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T14:32:49.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:32:49.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:32:49.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-08-19T14:32:50.768 INFO:tasks.workunit.client.0.smithi102.stdout:_ 2024-08-19T14:32:50.769 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 2024-08-19T14:32:51.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-08-19T14:32:51.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:32:51.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:32:51.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T14:32:51.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:32:51.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:32:51.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:32:51.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:32:51.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:32:51.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:32:51.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:32:51.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:32:51.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:32:51.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:32:51.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:32:51.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:32:51.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:32:51.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:32:51.038 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:32:51.038 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:32:51.038 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:32:51.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:32:51.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:32:51.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:32:51.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:32:51.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:32:51.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:32:51.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:32:51.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:32:51.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:32:51.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:32:51.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-08-19T14:32:51.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:32:51.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T14:32:51.040 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T14:32:51.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-08-19T14:32:51.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T14:32:51.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T14:32:51.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:32:51.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:32:51.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:32:51.069 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:51.068+0000 7fc8c774c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:32:51.072 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:51.070+0000 7fc8c774c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:32:51.073 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:51.071+0000 7fc8c774c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:32:51.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T14:32:51.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:32:51.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:32:51.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:32:51.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:32:51.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:32:51.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:32:51.320 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:32:51.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:32:51.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:32:51.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:32:51.599 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:51.598+0000 7fc8c774c640 -1 Falling back to public interface 2024-08-19T14:32:52.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:32:52.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:32:52.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:32:52.587 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:32:52.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:32:52.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:32:52.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:32:53.838 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:32:53.838+0000 7fc8c774c640 -1 osd.0 26 log_to_monitors true 2024-08-19T14:32:53.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:32:53.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:32:53.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:32:53.872 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:32:53.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:32:53.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:32:54.277 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/228725431,v1:127.0.0.1:6803/228725431] [v2:127.0.0.1:6804/228725431,v1:127.0.0.1:6805/228725431] exists,up 1f2360a0-17a2-47e8-83b2-fa195e397bf2 2024-08-19T14:32:54.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:32:54.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:32:54.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:32:54.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:32:54.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:32:54.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:32:54.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:32:54.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:32:54.278 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:32:54.278 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:32:54.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:32:54.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:32:54.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:32:54.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:32:54.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:32:54.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:32:54.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:32:54.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:32:54.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:32:54.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:32:54.740 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:32:54.740 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:32:54.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:32:54.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:32:54.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:32:54.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-08-19T14:32:54.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-08-19T14:32:54.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-08-19T14:32:54.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:32:54.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:32:55.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-08-19T14:32:55.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-08-19T14:32:55.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672975' 2024-08-19T14:32:55.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:32:55.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:32:55.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-08-19T14:32:55.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-08-19T14:32:55.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672975 2-64424509452' 2024-08-19T14:32:55.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:32:55.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-08-19T14:32:55.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:32:55.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:32:55.146 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-08-19T14:32:55.146 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:32:55.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-08-19T14:32:55.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-08-19T14:32:55.147 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 128849018882 2024-08-19T14:32:55.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:32:55.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-08-19T14:32:55.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:32:56.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:32:56.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:32:56.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-08-19T14:32:56.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:32:56.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-08-19T14:32:56.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:32:56.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:32:56.695 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-08-19T14:32:56.695 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:32:56.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-08-19T14:32:56.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-08-19T14:32:56.696 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949672975 2024-08-19T14:32:56.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:32:56.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-08-19T14:32:56.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:32:56.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-08-19T14:32:56.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:32:56.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:32:56.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-08-19T14:32:56.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:32:56.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-08-19T14:32:56.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-08-19T14:32:56.991 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509452 2024-08-19T14:32:56.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:32:57.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509452 2024-08-19T14:32:57.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:32:57.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:32:57.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:32:57.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:32:57.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:32:57.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:32:57.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:32:57.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:32:57.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:32:57.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:32:57.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:32:57.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:32:57.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:32:57.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:32:57.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:32:58.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:32:58.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:32:58.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:32:58.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-08-19T14:32:58.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-08-19T14:32:58.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-08-19T14:32:58.182 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-08-19T14:32:58.182 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T14:32:58.182 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:32:58.182 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:32:58.182 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:32:58.484 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T14:32:58.484 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-08-19T14:32:58.484 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T14:32:58.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-08-19T14:32:58.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:32:58.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:32:58.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T14:32:58.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:32:58.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-08-19T14:32:58.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:32:58.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:32:58.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T14:32:58.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:32:58.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T14:32:58.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:32:58.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:32:58.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:32:58.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:32:58.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:32:59.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:32:59.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-08-19T14:32:59.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:32:59.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:32:59.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T14:32:59.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:32:59.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:32:59.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-08-19T14:33:00.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-08-19T14:33:00.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:33:00.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:33:00.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T14:33:00.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:33:00.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:33:00.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:33:00.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:33:00.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:33:00.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:33:00.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:33:00.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:33:00.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:33:00.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:33:00.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:33:00.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:33:00.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:33:00.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:33:00.427 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:33:00.427 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:33:00.427 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:33:00.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:33:00.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:33:00.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:33:00.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:33:00.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:33:00.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:33:00.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:33:00.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:33:00.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:33:00.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:33:00.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-08-19T14:33:00.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:33:00.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T14:33:00.429 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T14:33:00.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-08-19T14:33:00.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T14:33:00.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T14:33:00.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:33:00.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:33:00.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:33:00.458 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:00.458+0000 7f5cf9a32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:33:00.461 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:00.461+0000 7f5cf9a32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:33:00.463 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:00.462+0000 7f5cf9a32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:33:00.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T14:33:00.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:33:00.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:33:00.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:33:00.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:33:00.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:00.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:33:00.710 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:33:00.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:00.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:00.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:00.982 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:00.982+0000 7f5cf9a32640 -1 Falling back to public interface 2024-08-19T14:33:01.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:01.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:01.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:33:01.977 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:33:01.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:01.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:02.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:03.215 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:03.215+0000 7f5cf9a32640 -1 osd.0 31 log_to_monitors true 2024-08-19T14:33:03.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:03.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:03.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:33:03.260 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:33:03.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:03.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:03.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:04.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:04.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:04.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:33:04.681 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:33:04.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:04.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:04.963 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/1971003755,v1:127.0.0.1:6803/1971003755] [v2:127.0.0.1:6804/1971003755,v1:127.0.0.1:6805/1971003755] exists,up 1f2360a0-17a2-47e8-83b2-fa195e397bf2 2024-08-19T14:33:04.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:33:04.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:33:04.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:33:04.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:33:04.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:33:04.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:33:04.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:33:04.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:33:04.964 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:33:04.964 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:33:04.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:33:04.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:33:04.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:33:05.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:33:05.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:33:05.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:33:05.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:33:05.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:33:05.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:33:05.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:33:05.321 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:33:05.321 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:33:05.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:33:05.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:33:05.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:33:05.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-08-19T14:33:05.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-08-19T14:33:05.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-08-19T14:33:05.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:33:05.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:33:05.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-08-19T14:33:05.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-08-19T14:33:05.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672978' 2024-08-19T14:33:05.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:33:05.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:33:05.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-08-19T14:33:05.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-08-19T14:33:05.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672978 2-64424509456' 2024-08-19T14:33:05.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:33:05.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-08-19T14:33:05.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:33:05.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:33:05.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-08-19T14:33:05.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:33:05.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-08-19T14:33:05.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-08-19T14:33:05.631 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 150323855362 2024-08-19T14:33:05.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:33:05.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-08-19T14:33:05.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:33:06.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:33:06.897 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:33:07.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-08-19T14:33:07.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:33:07.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-08-19T14:33:07.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:33:07.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:33:07.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-08-19T14:33:07.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:33:07.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-08-19T14:33:07.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-08-19T14:33:07.178 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949672978 2024-08-19T14:33:07.179 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:33:07.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-08-19T14:33:07.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:33:07.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-08-19T14:33:07.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:33:07.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:33:07.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-08-19T14:33:07.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:33:07.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-08-19T14:33:07.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-08-19T14:33:07.449 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509456 2024-08-19T14:33:07.449 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:33:07.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-08-19T14:33:07.717 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:33:07.717 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:33:07.717 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:33:08.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T14:33:08.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:33:08.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:33:08.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:33:08.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:33:08.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:33:08.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:33:08.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:33:08.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T14:33:08.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:33:08.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:33:08.322 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:33:08.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T14:33:08.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:33:08.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:33:08.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:432: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T14:33:08.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-08-19T14:33:08.659 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:32:46.201+0000 7f7c87921640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-08-19T14:33:08.659 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:32:46.201+0000 7f7c87921640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-08-19T14:33:08.659 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:32:46.201+0000 7f7c87921640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-08-19T14:33:08.659 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:32:46.201+0000 7f7c87921640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-08-19T14:33:08.659 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T14:32:46.201+0000 7f7c87921640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-08-19T14:33:08.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T14:33:08.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:33:08.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:33:08.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:33:08.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:33:08.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:33:08.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:33:08.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:33:08.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:33:08.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:33:08.767 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:33:08.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:33:08.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:33:08.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:33:08.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:33:08.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:33:08.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:33:08.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:33:08.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:33:08.772 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:33:08.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:33:08.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:33:08.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:33:08.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:33:08.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:33:08.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:33:08.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:33:08.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:33:08.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:33:08.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T14:33:08.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T14:33:08.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T14:33:08.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T14:33:08.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:33:08.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:33:08.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:33:08.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:33:08.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:33:08.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:33:08.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:33:08.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:33:08.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:33:08.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:33:08.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:33:08.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:33:08.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:33:08.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:33:08.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:33:08.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:33:08.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:33:08.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:33:08.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:33:08.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:33:08.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:33:08.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:33:08.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:33:08.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:33:08.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:33:08.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:33:08.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:33:08.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:33:08.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T14:33:08.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T14:33:08.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:33:08.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:33:08.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T14:33:08.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T14:33:08.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T14:33:08.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T14:33:08.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T14:33:08.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2024-08-19T14:33:08.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2024-08-19T14:33:08.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-08-19T14:33:08.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=false 2024-08-19T14:33:08.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-08-19T14:33:08.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-08-19T14:33:08.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T14:33:08.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T14:33:08.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T14:33:08.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T14:33:08.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T14:33:08.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-08-19T14:33:09.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T14:33:09.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:33:09.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:33:09.190 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:33:09.191 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:33:09.191 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:33:09.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:33:09.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T14:33:09.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T14:33:09.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T14:33:09.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:33:09.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:33:09.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T14:33:09.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T14:33:09.352 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:33:09.352 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:33:09.352 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:33:09.352 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:33:09.353 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:33:09.353 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:33:09.353 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:33:09.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:33:09.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T14:33:09.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T14:33:09.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:33:09.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:33:09.433 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T14:33:09.433 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T14:33:09.433 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:33:09.433 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:33:09.433 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:33:09.433 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:33:09.434 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:33:09.434 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:33:09.434 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:33:09.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:33:09.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T14:33:09.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-08-19T14:33:09.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T14:33:09.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T14:33:09.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T14:33:09.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T14:33:09.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T14:33:09.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T14:33:10.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T14:33:10.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:33:10.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:33:10.317 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:33:10.317 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:33:10.317 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:33:10.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:33:10.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:33:10.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:33:10.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:33:10.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-08-19T14:33:10.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-08-19T14:33:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:33:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:33:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:33:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T14:33:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:33:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:33:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:33:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:33:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:33:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:33:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:33:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:33:10.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:33:10.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:33:10.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:33:10.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:33:10.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:33:10.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:33:10.351 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:33:10.351 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:33:10.351 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:33:10.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:33:10.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:33:10.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:33:10.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:33:10.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:33:10.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:33:10.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:33:10.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:33:10.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:33:10.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:33:10.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:33:10.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:33:10.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:33:10.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:33:10.354 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:33:10.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ef18dab9-b400-43be-8c8b-7afceff56056 2024-08-19T14:33:10.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ef18dab9-b400-43be-8c8b-7afceff56056' 2024-08-19T14:33:10.354 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 ef18dab9-b400-43be-8c8b-7afceff56056 2024-08-19T14:33:10.354 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:33:10.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCmV8NmNvTnFRAAqw3GJAkODmjHKJIlPHTFxw== 2024-08-19T14:33:10.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCmV8NmNvTnFRAAqw3GJAkODmjHKJIlPHTFxw=="}' 2024-08-19T14:33:10.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ef18dab9-b400-43be-8c8b-7afceff56056 -i td/osd-scrub-repair/0/new.json 2024-08-19T14:33:10.565 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:33:10.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T14:33:10.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCmV8NmNvTnFRAAqw3GJAkODmjHKJIlPHTFxw== --osd-uuid ef18dab9-b400-43be-8c8b-7afceff56056 2024-08-19T14:33:10.607 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:10.607+0000 7f307101d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:33:10.616 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:10.615+0000 7f307101d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:33:10.617 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:10.616+0000 7f307101d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:33:10.618 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:10.617+0000 7f307101d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:33:10.618 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:10.617+0000 7f307101d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:33:10.618 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:10.617+0000 7f307101d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:33:10.707 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:10.707+0000 7f307101d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T14:33:16.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T14:33:16.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:33:16.156 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T14:33:16.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T14:33:16.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:33:16.511 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T14:33:16.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T14:33:16.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:33:16.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:33:16.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:33:16.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:33:16.542 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:16.541+0000 7ffabdead640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:33:16.543 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:16.543+0000 7ffabdead640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:33:16.545 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:16.544+0000 7ffabdead640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:33:16.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T14:33:16.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:33:16.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:33:16.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:33:16.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:33:16.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:16.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:33:16.779 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:33:16.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:16.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:17.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:17.063 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:17.063+0000 7ffabdead640 -1 Falling back to public interface 2024-08-19T14:33:18.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:18.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:18.045 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:33:18.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:33:18.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:18.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:18.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:19.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:19.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:19.316 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:33:19.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:33:19.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:19.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:19.357 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:19.356+0000 7ffabdead640 -1 osd.0 0 log_to_monitors true 2024-08-19T14:33:19.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:20.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:20.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:20.607 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:33:20.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:33:20.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:20.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:20.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:21.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:21.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:21.887 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:33:21.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:33:21.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:21.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:22.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:23.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:23.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:23.233 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:33:23.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:33:23.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:23.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:23.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:24.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:24.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:24.505 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:33:24.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:33:24.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:24.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:24.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:25.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:25.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:25.781 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:33:25.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:33:25.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:25.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:26.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:27.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:27.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:27.078 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:33:27.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:33:27.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:27.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:27.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:28.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:28.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:28.355 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:33:28.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:33:28.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:28.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:28.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:29.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:29.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:29.632 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:33:29.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:33:29.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:29.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:30.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:31.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:31.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:31.114 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:33:31.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:33:31.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:31.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:31.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:32.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:32.744 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:33:32.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:32.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:33:32.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:32.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:33.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:34.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:34.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:34.288 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:33:34.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:33:34.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:34.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:35.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:36.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:36.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:36.141 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:33:36.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:33:36.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:36.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:36.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:37.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:37.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:37.577 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:33:37.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:33:37.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:37.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:38.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:39.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:39.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:39.084 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:33:39.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:33:39.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:39.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:39.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:40.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:40.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:40.360 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:33:40.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:33:40.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:40.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:40.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:41.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:41.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:41.810 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:33:41.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:33:41.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:41.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:33:42.081 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/207694269,v1:127.0.0.1:6803/207694269] [v2:127.0.0.1:6804/207694269,v1:127.0.0.1:6805/207694269] exists,up ef18dab9-b400-43be-8c8b-7afceff56056 2024-08-19T14:33:42.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:33:42.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:33:42.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:33:42.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-08-19T14:33:42.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:33:42.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:33:42.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:33:42.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T14:33:42.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:33:42.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:33:42.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:33:42.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:33:42.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:33:42.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:33:42.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:33:42.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:33:42.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:33:42.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:33:42.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:33:42.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:33:42.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:33:42.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:33:42.087 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:33:42.087 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:33:42.088 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:33:42.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:33:42.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:33:42.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:33:42.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:33:42.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:33:42.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:33:42.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:33:42.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:33:42.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:33:42.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:33:42.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:33:42.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:33:42.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:33:42.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:33:42.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:33:42.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8a341c8d-910a-496a-8bd3-573b8924d275 2024-08-19T14:33:42.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8a341c8d-910a-496a-8bd3-573b8924d275' 2024-08-19T14:33:42.089 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 8a341c8d-910a-496a-8bd3-573b8924d275 2024-08-19T14:33:42.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:33:42.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDGV8NmnmQpBhAAUnt8m7yDzzluW3M9jHXKDA== 2024-08-19T14:33:42.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDGV8NmnmQpBhAAUnt8m7yDzzluW3M9jHXKDA=="}' 2024-08-19T14:33:42.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8a341c8d-910a-496a-8bd3-573b8924d275 -i td/osd-scrub-repair/1/new.json 2024-08-19T14:33:42.442 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:33:42.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T14:33:42.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDGV8NmnmQpBhAAUnt8m7yDzzluW3M9jHXKDA== --osd-uuid 8a341c8d-910a-496a-8bd3-573b8924d275 2024-08-19T14:33:42.487 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:42.486+0000 7fa4dbb1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:33:42.519 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:42.518+0000 7fa4dbb1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:33:42.520 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:42.519+0000 7fa4dbb1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:33:42.520 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:42.520+0000 7fa4dbb1d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:33:42.520 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:42.520+0000 7fa4dbb1d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:33:42.520 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:42.520+0000 7fa4dbb1d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:33:42.595 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:42.594+0000 7fa4dbb1d640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T14:33:48.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T14:33:48.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:33:48.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T14:33:48.255 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T14:33:48.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:33:48.600 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T14:33:48.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T14:33:48.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:33:48.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:33:48.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:33:48.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:33:48.630 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:48.630+0000 7f8bc080d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:33:48.632 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:48.632+0000 7f8bc080d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:33:48.633 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:48.633+0000 7f8bc080d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:33:48.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T14:33:48.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:33:48.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:33:48.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:33:48.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:33:48.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:48.870 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:33:48.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:33:48.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:48.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:33:49.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:49.165 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:49.165+0000 7f8bc080d640 -1 Falling back to public interface 2024-08-19T14:33:50.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:50.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:50.139 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:33:50.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:33:50.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:50.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:33:50.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:51.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:51.424 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:33:51.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:51.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:33:51.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:51.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:33:51.478 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:33:51.477+0000 7f8bc080d640 -1 osd.1 0 log_to_monitors true 2024-08-19T14:33:51.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:52.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:52.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:52.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:33:52.699 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:33:52.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:52.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:33:52.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:53.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:53.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:53.973 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:33:53.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:33:53.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:53.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:33:54.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:55.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:55.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:55.318 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:33:55.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:33:55.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:55.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:33:55.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:56.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:56.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:56.594 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:33:56.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:33:56.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:56.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:33:56.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:57.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:57.920 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:33:57.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:57.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:33:57.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:57.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:33:58.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:33:59.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:33:59.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:33:59.257 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:33:59.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:33:59.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:33:59.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:33:59.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:00.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:00.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:00.534 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:34:00.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:34:00.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:00.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:34:00.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:01.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:01.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:01.935 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:34:01.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:34:01.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:01.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:34:02.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:03.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:03.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:03.287 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:34:03.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:34:03.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:03.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:34:03.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:04.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:04.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:04.890 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:34:04.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:34:04.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:04.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:34:05.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:06.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:06.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:06.635 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:34:06.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:34:06.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:06.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:34:07.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:08.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:08.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:08.321 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:34:08.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:34:08.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:08.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:34:08.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:09.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:09.671 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:34:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:34:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:34:10.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:11.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:11.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:11.188 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:34:11.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:34:11.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:11.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:34:11.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:12.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:12.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:12.544 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:34:12.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:34:12.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:12.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:34:12.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:13.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:13.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:13.996 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:34:13.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:34:13.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:13.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:34:14.309 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3336804464,v1:127.0.0.1:6811/3336804464] [v2:127.0.0.1:6812/3336804464,v1:127.0.0.1:6813/3336804464] exists,up 8a341c8d-910a-496a-8bd3-573b8924d275 2024-08-19T14:34:14.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:34:14.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:34:14.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:34:14.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-08-19T14:34:14.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:34:14.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:34:14.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:34:14.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T14:34:14.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:34:14.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T14:34:14.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:34:14.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:34:14.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:34:14.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:34:14.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T14:34:14.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T14:34:14.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:34:14.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:34:14.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:34:14.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:34:14.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:34:14.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:34:14.314 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:34:14.314 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:34:14.314 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:34:14.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:34:14.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:34:14.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:34:14.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:34:14.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:34:14.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:34:14.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:34:14.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:34:14.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:34:14.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:34:14.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:34:14.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:34:14.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:34:14.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T14:34:14.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:34:14.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=02c77a01-c5f2-473a-9c00-11263f56a8f5 2024-08-19T14:34:14.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 02c77a01-c5f2-473a-9c00-11263f56a8f5' 2024-08-19T14:34:14.316 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 02c77a01-c5f2-473a-9c00-11263f56a8f5 2024-08-19T14:34:14.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:34:14.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDmV8Nm5KO7ExAAq/WyBAr7TOkE52IguoKNFQ== 2024-08-19T14:34:14.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDmV8Nm5KO7ExAAq/WyBAr7TOkE52IguoKNFQ=="}' 2024-08-19T14:34:14.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 02c77a01-c5f2-473a-9c00-11263f56a8f5 -i td/osd-scrub-repair/2/new.json 2024-08-19T14:34:14.675 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:34:14.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-08-19T14:34:14.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDmV8Nm5KO7ExAAq/WyBAr7TOkE52IguoKNFQ== --osd-uuid 02c77a01-c5f2-473a-9c00-11263f56a8f5 2024-08-19T14:34:14.718 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:34:14.718+0000 7f1ae540d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:34:14.760 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:34:14.759+0000 7f1ae540d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:34:14.761 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:34:14.760+0000 7f1ae540d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:34:14.761 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:34:14.761+0000 7f1ae540d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:34:14.761 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:34:14.761+0000 7f1ae540d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:34:14.761 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:34:14.761+0000 7f1ae540d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:34:14.826 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:34:14.826+0000 7f1ae540d640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-08-19T14:34:20.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-08-19T14:34:20.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:34:20.938 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T14:34:20.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T14:34:20.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:34:21.290 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T14:34:21.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T14:34:21.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:34:21.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:34:21.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:34:21.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:34:21.321 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:34:21.320+0000 7fb166bc4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:34:21.322 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:34:21.322+0000 7fb166bc4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:34:21.323 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:34:21.323+0000 7fb166bc4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:34:21.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T14:34:21.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:34:21.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T14:34:21.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:34:21.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:34:21.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:21.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:34:21.559 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:34:21.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:21.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:21.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:21.844 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:34:21.844+0000 7fb166bc4640 -1 Falling back to public interface 2024-08-19T14:34:22.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:22.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:22.826 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:34:22.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:34:22.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:22.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:23.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:24.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:24.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:24.098 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:34:24.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:34:24.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:24.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:24.214 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:34:24.212+0000 7fb166bc4640 -1 osd.2 0 log_to_monitors true 2024-08-19T14:34:24.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:25.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:25.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:25.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:34:25.377 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:34:25.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:25.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:25.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:26.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:26.697 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:34:26.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:26.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:34:26.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:26.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:27.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:28.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:28.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:28.049 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:34:28.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:34:28.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:28.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:28.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:29.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:29.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:29.327 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:34:29.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:34:29.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:29.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:29.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:30.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:30.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:30.607 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:34:30.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:34:30.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:30.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:30.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:31.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:31.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:31.978 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:34:31.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:34:31.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:31.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:32.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:33.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:33.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:33.320 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:34:33.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:34:33.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:33.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:33.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:34.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:34.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:34.598 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:34:34.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:34:34.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:34.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:34.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:35.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:35.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:35.885 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:34:35.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:34:35.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:35.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:36.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:37.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:37.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:37.227 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:34:37.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:34:37.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:37.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:37.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:38.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:38.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:38.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:34:38.704 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:34:38.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:38.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:39.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:40.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:40.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:40.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:34:40.176 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:34:40.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:40.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:40.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:41.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:41.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:41.589 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:34:41.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:34:41.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:41.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:42.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:43.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:43.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:43.146 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:34:43.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:34:43.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:43.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:43.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:44.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:44.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:44.671 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:34:44.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:34:44.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:44.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:45.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:46.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:46.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:46.148 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:34:46.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:34:46.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:46.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:46.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:34:47.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:34:47.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:34:47.730 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T14:34:47.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T14:34:47.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:34:47.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:34:48.003 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3701831479,v1:127.0.0.1:6819/3701831479] [v2:127.0.0.1:6820/3701831479,v1:127.0.0.1:6821/3701831479] exists,up 02c77a01-c5f2-473a-9c00-11263f56a8f5 2024-08-19T14:34:48.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:34:48.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:34:48.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:34:48.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-08-19T14:34:48.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T14:34:48.305 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-08-19T14:34:48.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T14:34:48.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T14:34:48.763 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' created 2024-08-19T14:34:48.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T14:34:49.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T14:34:51.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-08-19T14:34:51.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:34:51.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:34:51.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:34:51.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:34:51.520 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:34:51.520 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:34:51.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:34:51.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:34:51.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:34:51.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:34:51.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:34:51.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:34:51.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:34:51.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:34:51.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:34:51.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:34:51.900 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:34:51.900 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:34:51.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:34:51.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:34:51.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:34:52.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-08-19T14:34:52.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-08-19T14:34:52.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-08-19T14:34:52.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:34:52.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:34:52.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-08-19T14:34:52.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-08-19T14:34:52.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-64424509458' 2024-08-19T14:34:52.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:34:52.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:34:52.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051590 2024-08-19T14:34:52.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051590 2024-08-19T14:34:52.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-64424509458 2-124554051590' 2024-08-19T14:34:52.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:34:52.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-08-19T14:34:52.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:34:52.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:34:52.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-08-19T14:34:52.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:34:52.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-08-19T14:34:52.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-08-19T14:34:52.212 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836501 2024-08-19T14:34:52.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:34:52.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836501 2024-08-19T14:34:52.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:34:52.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-64424509458 2024-08-19T14:34:52.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:34:52.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:34:52.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-64424509458 2024-08-19T14:34:52.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:34:52.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-08-19T14:34:52.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 64424509458' 2024-08-19T14:34:52.485 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 64424509458 2024-08-19T14:34:52.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:34:52.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-08-19T14:34:52.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:34:52.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-124554051590 2024-08-19T14:34:52.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:34:52.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:34:52.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-124554051590 2024-08-19T14:34:52.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:34:52.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051590 2024-08-19T14:34:52.798 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 124554051590 2024-08-19T14:34:52.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 124554051590' 2024-08-19T14:34:52.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:34:53.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051589 -lt 124554051590 2024-08-19T14:34:53.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:34:54.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:34:54.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:34:54.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051591 -lt 124554051590 2024-08-19T14:34:54.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:34:54.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:34:54.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:34:54.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T14:34:54.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:34:54.679 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:34:54.679 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:34:54.679 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:34:54.679 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:34:54.679 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:34:54.679 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:34:54.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:34:54.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:34:54.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:34:54.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:34:55.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T14:34:55.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:34:55.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:34:55.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-08-19T14:34:55.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-08-19T14:34:55.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-08-19T14:34:55.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-08-19T14:34:55.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-08-19T14:34:55.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-08-19T14:34:55.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-08-19T14:34:55.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-08-19T14:34:56.475 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'ecpool' created 2024-08-19T14:34:56.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T14:34:57.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-08-19T14:34:57.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-08-19T14:34:57.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:34:57.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:34:57.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:34:57.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:34:57.490 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:34:57.490 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:34:57.490 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:34:57.490 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:34:57.490 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:34:57.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:34:57.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:34:57.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:34:57.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:34:57.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:34:57.591 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:34:57.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:34:57.912 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:34:57.912 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:34:57.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:34:57.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:34:57.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:34:58.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-08-19T14:34:58.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-08-19T14:34:58.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-08-19T14:34:58.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:34:58.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:34:58.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-08-19T14:34:58.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-08-19T14:34:58.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-64424509460' 2024-08-19T14:34:58.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:34:58.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:34:58.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051593 2024-08-19T14:34:58.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051593 2024-08-19T14:34:58.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-64424509460 2-124554051593' 2024-08-19T14:34:58.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:34:58.322 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-08-19T14:34:58.322 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:34:58.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:34:58.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-08-19T14:34:58.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:34:58.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-08-19T14:34:58.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-08-19T14:34:58.324 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836503 2024-08-19T14:34:58.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:34:58.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-08-19T14:34:58.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:34:58.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-64424509460 2024-08-19T14:34:58.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:34:58.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:34:58.616 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-64424509460 2024-08-19T14:34:58.617 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:34:58.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-08-19T14:34:58.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 64424509460' 2024-08-19T14:34:58.618 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 64424509460 2024-08-19T14:34:58.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:34:58.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-08-19T14:34:58.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:34:58.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-124554051593 2024-08-19T14:34:58.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:34:58.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:34:58.885 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-124554051593 2024-08-19T14:34:58.885 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:34:58.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051593 2024-08-19T14:34:58.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 124554051593' 2024-08-19T14:34:58.886 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 124554051593 2024-08-19T14:34:58.886 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:34:59.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051592 -lt 124554051593 2024-08-19T14:34:59.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:35:00.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:35:00.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:35:00.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051593 -lt 124554051593 2024-08-19T14:35:00.430 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:35:00.430 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:35:00.430 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:35:00.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:35:00.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:35:00.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:35:00.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:35:00.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:35:00.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:35:00.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:35:00.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:35:01.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:35:01.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:35:01.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:35:01.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:35:01.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:35:01.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:35:01.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:35:01.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-08-19T14:35:01.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-08-19T14:35:01.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-08-19T14:35:01.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-08-19T14:35:01.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-08-19T14:35:01.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-08-19T14:35:01.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-08-19T14:35:01.509 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-08-19T14:35:01.509 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-08-19T14:35:01.509 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:35:01.509 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:35:01.509 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:35:01.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-08-19T14:35:01.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:35:01.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-08-19T14:35:02.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:35:02.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:35:02.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:35:02.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T14:35:02.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:35:02.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:35:02.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:35:02.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:35:02.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T14:35:02.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:35:02.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T14:35:02.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:35:02.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:35:02.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:35:02.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:35:02.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:35:02.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:35:02.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:35:02.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:35:02.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:35:02.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T14:35:02.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:35:02.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:35:02.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-08-19T14:35:04.281 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-08-19T14:35:04.281 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-08-19T14:35:04.282 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-08-19T14:35:04.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:35:04.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:35:04.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:35:04.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:35:04.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:35:04.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:35:04.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:35:04.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:35:04.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:35:04.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:35:04.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:35:04.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:35:04.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:35:04.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:35:04.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:35:04.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:35:04.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:35:04.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:35:04.803 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:35:04.803 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:35:04.803 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:35:04.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:35:04.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:35:04.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:35:04.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:35:04.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:35:04.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:35:04.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:35:04.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:35:04.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:35:04.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:35:04.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:35:04.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:35:04.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:35:04.805 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T14:35:04.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:35:04.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T14:35:04.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:35:04.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:35:04.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:35:04.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:35:04.836 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:04.835+0000 7f9fb19c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:35:04.838 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:04.838+0000 7f9fb19c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:35:04.839 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:04.839+0000 7f9fb19c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:35:05.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:35:05.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:35:05.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:35:05.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:35:05.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:35:05.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:35:05.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:35:05.081 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:35:05.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:35:05.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:35:05.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:35:05.357 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:05.357+0000 7f9fb19c0640 -1 Falling back to public interface 2024-08-19T14:35:06.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:35:06.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:35:06.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:35:06.346 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:35:06.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:35:06.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:35:06.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:35:07.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:35:07.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:35:07.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:35:07.628 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:35:07.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:35:07.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:35:07.720 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:07.719+0000 7f9fb19c0640 -1 osd.1 49 log_to_monitors true 2024-08-19T14:35:07.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:35:08.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:35:08.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:35:08.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:35:08.976 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:35:08.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:35:08.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:35:09.409 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 50 last_clean_interval [15,49) [v2:127.0.0.1:6810/1070185986,v1:127.0.0.1:6811/1070185986] [v2:127.0.0.1:6812/1070185986,v1:127.0.0.1:6813/1070185986] exists,up 8a341c8d-910a-496a-8bd3-573b8924d275 2024-08-19T14:35:09.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:35:09.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:35:09.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:35:09.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:35:09.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:35:09.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:35:09.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:35:09.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:35:09.410 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:35:09.410 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:35:09.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:35:09.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:35:09.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:35:09.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:35:09.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:35:09.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:35:09.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:35:09.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:35:09.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:35:09.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:35:09.829 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:35:09.829 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:35:09.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:35:09.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:35:09.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:35:10.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-08-19T14:35:10.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-08-19T14:35:10.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-08-19T14:35:10.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:35:10.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:35:10.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135874 2024-08-19T14:35:10.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135874 2024-08-19T14:35:10.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-244813135874' 2024-08-19T14:35:10.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:35:10.223 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:35:10.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051596 2024-08-19T14:35:10.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051596 2024-08-19T14:35:10.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-244813135874 2-124554051596' 2024-08-19T14:35:10.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:35:10.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-08-19T14:35:10.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:35:10.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:35:10.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-08-19T14:35:10.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:35:10.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-08-19T14:35:10.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-08-19T14:35:10.391 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836506 2024-08-19T14:35:10.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:35:10.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836506 2024-08-19T14:35:10.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:35:10.679 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135874 2024-08-19T14:35:10.679 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:35:10.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:35:10.680 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135874 2024-08-19T14:35:10.680 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:35:10.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135874 2024-08-19T14:35:10.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-08-19T14:35:10.682 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 244813135874 2024-08-19T14:35:10.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:35:10.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 244813135874 2024-08-19T14:35:10.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:35:11.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:35:11.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:35:12.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 244813135874 2024-08-19T14:35:12.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:35:13.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T14:35:13.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:35:13.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135874 -lt 244813135874 2024-08-19T14:35:13.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:35:13.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-124554051596 2024-08-19T14:35:13.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:35:13.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:35:13.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-124554051596 2024-08-19T14:35:13.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:35:13.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051596 2024-08-19T14:35:13.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 124554051596' 2024-08-19T14:35:13.518 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 124554051596 2024-08-19T14:35:13.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:35:13.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051596 -lt 124554051596 2024-08-19T14:35:13.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:35:13.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:35:13.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:35:14.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:35:14.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:35:14.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:35:14.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:35:14.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:35:14.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:35:14.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:35:14.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:35:14.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:35:14.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:35:14.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:35:14.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:35:14.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:35:14.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:35:14.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:35:14.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-08-19T14:35:14.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T14:35:14.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T14:35:14.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:35:14.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:35:15.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-08-19T14:35:15.195 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-08-19T14:35:15.195 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:35:15.195 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:35:15.195 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:35:15.195 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:35:15.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-08-19T14:35:11.190366+0000 2024-08-19T14:35:15.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T14:35:15.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:35:11.190366+0000 2024-08-19T14:35:15.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:35:15.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:35:15.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:35:15.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:35:15.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:35:15.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:35:15.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:35:15.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:35:15.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:35:11.190366+0000 '>' 2024-08-19T14:35:11.190366+0000 2024-08-19T14:35:15.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:35:16.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:35:16.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:35:16.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:35:16.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:35:16.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:35:16.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:35:16.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:35:16.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:35:11.190366+0000 '>' 2024-08-19T14:35:11.190366+0000 2024-08-19T14:35:16.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:35:17.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:35:17.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:35:17.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:35:17.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:35:17.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:35:17.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:35:17.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:35:18.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:35:11.190366+0000 '>' 2024-08-19T14:35:11.190366+0000 2024-08-19T14:35:18.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:35:19.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:35:19.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:35:19.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:35:19.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:35:19.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:35:19.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:35:19.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:35:19.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:35:11.190366+0000 '>' 2024-08-19T14:35:11.190366+0000 2024-08-19T14:35:19.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:35:20.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:35:20.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:35:20.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:35:20.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:35:20.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:35:20.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:35:20.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:35:20.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:35:11.190366+0000 '>' 2024-08-19T14:35:11.190366+0000 2024-08-19T14:35:20.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:35:21.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:35:21.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:35:21.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:35:21.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:35:21.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:35:21.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:35:21.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:35:22.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:35:11.190366+0000 '>' 2024-08-19T14:35:11.190366+0000 2024-08-19T14:35:22.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:35:23.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:35:23.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:35:23.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:35:23.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:35:23.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:35:23.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:35:23.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:35:23.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:35:11.190366+0000 '>' 2024-08-19T14:35:11.190366+0000 2024-08-19T14:35:23.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:35:24.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:35:24.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:35:24.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:35:24.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:35:24.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:35:24.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:35:24.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:35:24.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:35:11.190366+0000 '>' 2024-08-19T14:35:11.190366+0000 2024-08-19T14:35:24.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:35:25.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:35:25.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:35:25.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:35:25.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:35:25.631 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:35:25.631 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:35:25.631 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:35:25.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:35:11.190366+0000 '>' 2024-08-19T14:35:11.190366+0000 2024-08-19T14:35:25.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:35:26.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:35:26.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:35:26.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:35:26.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:35:26.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:35:26.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:35:26.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:35:27.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:35:11.190366+0000 '>' 2024-08-19T14:35:11.190366+0000 2024-08-19T14:35:27.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:35:28.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:35:28.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:35:28.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:35:28.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:35:28.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:35:28.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:35:28.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:35:28.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:35:25.450976+0000 '>' 2024-08-19T14:35:11.190366+0000 2024-08-19T14:35:28.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:35:28.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-08-19T14:35:28.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:35:28.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:35:28.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:35:28.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:35:28.468 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:35:28.468 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:35:28.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:35:28.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:35:28.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:35:28.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:35:28.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:35:28.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:35:28.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:35:28.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:35:28.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:35:28.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:35:28.824 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:35:28.825 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:35:28.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:35:28.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:35:28.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:35:28.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836511 2024-08-19T14:35:28.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836511 2024-08-19T14:35:28.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511' 2024-08-19T14:35:28.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:35:28.928 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:35:29.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135879 2024-08-19T14:35:29.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135879 2024-08-19T14:35:29.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-244813135879' 2024-08-19T14:35:29.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:35:29.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:35:29.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051601 2024-08-19T14:35:29.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051601 2024-08-19T14:35:29.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-244813135879 2-124554051601' 2024-08-19T14:35:29.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:35:29.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836511 2024-08-19T14:35:29.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:35:29.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:35:29.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836511 2024-08-19T14:35:29.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:35:29.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836511 2024-08-19T14:35:29.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-08-19T14:35:29.178 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836511 2024-08-19T14:35:29.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:35:29.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836511 2024-08-19T14:35:29.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:35:30.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:35:30.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:35:30.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836511 2024-08-19T14:35:30.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:35:30.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135879 2024-08-19T14:35:30.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:35:30.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:35:30.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135879 2024-08-19T14:35:30.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:35:30.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135879 2024-08-19T14:35:30.750 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 244813135879 2024-08-19T14:35:30.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135879' 2024-08-19T14:35:30.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:35:31.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135879 -lt 244813135879 2024-08-19T14:35:31.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:35:31.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-124554051601 2024-08-19T14:35:31.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:35:31.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:35:31.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-124554051601 2024-08-19T14:35:31.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:35:31.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051601 2024-08-19T14:35:31.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 124554051601' 2024-08-19T14:35:31.020 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 124554051601 2024-08-19T14:35:31.020 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:35:31.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051601 -lt 124554051601 2024-08-19T14:35:31.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:35:31.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:35:31.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:35:31.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:35:31.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:35:31.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:35:31.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:35:31.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:35:31.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:35:31.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:35:31.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:35:31.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:35:31.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:35:31.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:35:31.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:35:32.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:35:32.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:35:32.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:35:32.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-08-19T14:35:32.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-08-19T14:35:32.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-08-19T14:35:32.211 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-08-19T14:35:32.211 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-08-19T14:35:32.211 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:35:32.211 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:35:32.211 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:35:32.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-08-19T14:35:32.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:35:32.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-08-19T14:35:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:35:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:35:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:35:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T14:35:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:35:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:35:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:35:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:35:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T14:35:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:35:32.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T14:35:32.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:35:32.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:35:32.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:35:32.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:35:32.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:35:33.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:35:33.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:35:33.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:35:33.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:35:33.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T14:35:33.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:35:33.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:35:33.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-08-19T14:35:34.753 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-08-19T14:35:34.754 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-08-19T14:35:34.754 INFO:tasks.workunit.client.0.smithi102.stdout:_ 2024-08-19T14:35:34.754 INFO:tasks.workunit.client.0.smithi102.stdout:hinfo_key 2024-08-19T14:35:34.754 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 2024-08-19T14:35:35.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:35:35.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:35:35.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:35:35.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:35:35.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:35:35.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:35:35.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:35:35.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:35:35.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:35:35.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:35:35.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:35:35.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:35:35.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:35:35.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:35:35.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:35:35.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:35:35.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:35:35.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:35:35.274 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:35:35.274 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:35:35.274 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:35:35.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:35:35.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:35:35.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:35:35.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:35:35.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:35:35.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:35:35.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:35:35.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:35:35.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:35:35.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:35:35.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:35:35.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:35:35.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:35:35.275 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T14:35:35.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:35:35.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T14:35:35.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:35:35.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:35:35.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:35:35.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:35:35.306 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:35.306+0000 7f81b7e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:35:35.309 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:35.308+0000 7f81b7e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:35:35.310 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:35.310+0000 7f81b7e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:35:35.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:35:35.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:35:35.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:35:35.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:35:35.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:35:35.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:35:35.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:35:35.570 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:35:35.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:35:35.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:35:35.824 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:35.824+0000 7f81b7e18640 -1 Falling back to public interface 2024-08-19T14:35:35.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:35:36.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:35:36.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:35:36.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:35:36.840 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:35:36.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:35:36.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:35:37.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:35:38.056 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:38.055+0000 7f81b7e18640 -1 osd.1 73 log_to_monitors true 2024-08-19T14:35:38.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:35:38.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:35:38.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:35:38.114 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:35:38.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:35:38.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:35:38.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:35:39.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:35:39.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:35:39.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:35:39.445 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:35:39.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:35:39.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:35:39.786 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 80 up_thru 80 down_at 74 last_clean_interval [57,73) [v2:127.0.0.1:6810/1797394072,v1:127.0.0.1:6811/1797394072] [v2:127.0.0.1:6812/1797394072,v1:127.0.0.1:6813/1797394072] exists,up 8a341c8d-910a-496a-8bd3-573b8924d275 2024-08-19T14:35:39.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:35:39.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:35:39.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:35:39.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:35:39.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:35:39.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:35:39.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:35:39.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:35:39.787 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:35:39.787 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:35:39.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:35:39.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:35:39.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:35:39.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:35:39.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:35:39.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:35:39.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:35:39.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:35:39.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:35:40.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:35:40.159 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:35:40.159 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:35:40.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:35:40.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:35:40.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:35:40.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836514 2024-08-19T14:35:40.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836514 2024-08-19T14:35:40.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514' 2024-08-19T14:35:40.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:35:40.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:35:40.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383682 2024-08-19T14:35:40.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383682 2024-08-19T14:35:40.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-343597383682' 2024-08-19T14:35:40.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:35:40.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:35:40.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051604 2024-08-19T14:35:40.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051604 2024-08-19T14:35:40.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-343597383682 2-124554051604' 2024-08-19T14:35:40.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:35:40.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836514 2024-08-19T14:35:40.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:35:40.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:35:40.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836514 2024-08-19T14:35:40.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:35:40.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836514 2024-08-19T14:35:40.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-08-19T14:35:40.526 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836514 2024-08-19T14:35:40.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:35:40.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836514 2024-08-19T14:35:40.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:35:41.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:35:41.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:35:42.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836514 2024-08-19T14:35:42.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:35:43.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T14:35:43.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:35:43.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836515 -lt 21474836514 2024-08-19T14:35:43.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:35:43.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-343597383682 2024-08-19T14:35:43.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:35:43.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:35:43.377 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-343597383682 2024-08-19T14:35:43.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:35:43.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383682 2024-08-19T14:35:43.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 343597383682' 2024-08-19T14:35:43.379 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 343597383682 2024-08-19T14:35:43.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:35:43.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383682 -lt 343597383682 2024-08-19T14:35:43.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:35:43.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-124554051604 2024-08-19T14:35:43.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:35:43.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:35:43.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-124554051604 2024-08-19T14:35:43.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:35:43.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051604 2024-08-19T14:35:43.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 124554051604' 2024-08-19T14:35:43.653 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 124554051604 2024-08-19T14:35:43.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:35:43.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051604 -lt 124554051604 2024-08-19T14:35:43.922 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:35:43.922 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:35:43.922 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:35:44.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:35:44.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:35:44.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:35:44.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:35:44.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:35:44.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:35:44.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:35:44.339 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:35:44.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:35:44.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:35:44.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:35:44.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:35:44.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:35:44.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:35:44.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:35:44.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T14:35:45.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T14:35:45.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T14:35:45.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:35:45.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:35:45.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:35:45.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:35:45.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:35:45.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:35:45.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:35:45.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:35:45.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:35:45.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:35:45.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:35:45.126 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:35:45.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:35:45.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:35:45.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:35:45.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:35:45.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:35:45.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:35:45.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:35:45.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:35:45.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:35:45.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:35:45.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:35:45.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:35:45.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:35:45.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:35:45.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:35:45.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:35:45.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T14:35:45.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T14:35:45.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T14:35:45.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T14:35:45.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:35:45.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:35:45.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:35:45.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:35:45.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:35:45.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:35:45.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:35:45.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:35:45.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:35:45.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:35:45.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:35:45.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:35:45.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:35:45.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:35:45.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:35:45.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:35:45.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:35:45.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:35:45.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:35:45.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:35:45.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:35:45.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:35:45.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:35:45.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:35:45.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:35:45.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:35:45.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:35:45.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:35:45.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T14:35:45.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T14:35:45.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:35:45.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:35:45.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T14:35:45.164 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T14:35:45.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T14:35:45.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T14:35:45.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T14:35:45.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2024-08-19T14:35:45.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2024-08-19T14:35:45.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2024-08-19T14:35:45.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-08-19T14:35:45.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=true 2024-08-19T14:35:45.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-08-19T14:35:45.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-08-19T14:35:45.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T14:35:45.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T14:35:45.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T14:35:45.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T14:35:45.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T14:35:45.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-08-19T14:35:45.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T14:35:45.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:35:45.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:35:45.527 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:35:45.528 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:35:45.528 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:35:45.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:35:45.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T14:35:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T14:35:45.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T14:35:45.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:35:45.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:35:45.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T14:35:45.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T14:35:45.714 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:35:45.714 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:35:45.714 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:35:45.715 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:35:45.715 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:35:45.715 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:35:45.715 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:35:45.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:35:45.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T14:35:45.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T14:35:45.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:35:45.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:35:45.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T14:35:45.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T14:35:45.794 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:35:45.794 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:35:45.794 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:35:45.794 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:35:45.794 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:35:45.794 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:35:45.794 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:35:45.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:35:45.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T14:35:45.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-08-19T14:35:45.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T14:35:45.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T14:35:45.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T14:35:45.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T14:35:45.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T14:35:45.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T14:35:46.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T14:35:46.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:35:46.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:35:46.672 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:35:46.673 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:35:46.673 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:35:46.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:35:46.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:35:46.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:35:46.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:35:46.701 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-08-19T14:35:46.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-08-19T14:35:46.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:35:46.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:35:46.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:35:46.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T14:35:46.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:35:46.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:35:46.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:35:46.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:35:46.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:35:46.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:35:46.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:35:46.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:35:46.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:35:46.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:35:46.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:35:46.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:35:46.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:35:46.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:35:46.706 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:35:46.706 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:35:46.706 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:35:46.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:35:46.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:35:46.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:35:46.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:35:46.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:35:46.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:35:46.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:35:46.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:35:46.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:35:46.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:35:46.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:35:46.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:35:46.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:35:46.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:35:46.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:35:46.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=71f6b0bb-765a-46a1-b527-b89fce595765 2024-08-19T14:35:46.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 71f6b0bb-765a-46a1-b527-b89fce595765' 2024-08-19T14:35:46.708 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 71f6b0bb-765a-46a1-b527-b89fce595765 2024-08-19T14:35:46.709 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:35:46.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBCWMNmFbUaKxAAVW7Cnb633jqt5fPR+A9EXg== 2024-08-19T14:35:46.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBCWMNmFbUaKxAAVW7Cnb633jqt5fPR+A9EXg=="}' 2024-08-19T14:35:46.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 71f6b0bb-765a-46a1-b527-b89fce595765 -i td/osd-scrub-repair/0/new.json 2024-08-19T14:35:46.928 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:35:46.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T14:35:46.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBCWMNmFbUaKxAAVW7Cnb633jqt5fPR+A9EXg== --osd-uuid 71f6b0bb-765a-46a1-b527-b89fce595765 2024-08-19T14:35:46.971 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:46.970+0000 7f3269e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:35:46.980 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:46.979+0000 7f3269e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:35:46.981 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:46.980+0000 7f3269e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:35:46.981 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:46.981+0000 7f3269e0d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:35:46.981 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:46.981+0000 7f3269e0d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:35:46.981 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:46.981+0000 7f3269e0d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:35:47.088 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:47.087+0000 7f3269e0d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T14:35:52.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T14:35:52.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:35:52.561 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T14:35:52.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T14:35:52.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:35:52.912 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T14:35:52.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T14:35:52.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:35:52.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:35:52.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:35:52.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:35:52.943 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:52.942+0000 7f723de14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:35:52.945 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:52.944+0000 7f723de14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:35:52.946 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:52.946+0000 7f723de14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:35:53.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T14:35:53.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:35:53.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:35:53.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:35:53.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:35:53.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:35:53.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:35:53.181 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:35:53.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:35:53.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:35:53.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:35:53.475 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:53.475+0000 7f723de14640 -1 Falling back to public interface 2024-08-19T14:35:54.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:35:54.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:35:54.448 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:35:54.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:35:54.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:35:54.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:35:54.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:35:55.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:35:55.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:35:55.722 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:35:55.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:35:55.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:35:55.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:35:55.820 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:35:55.819+0000 7f723de14640 -1 osd.0 0 log_to_monitors true 2024-08-19T14:35:55.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:35:56.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:35:56.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:35:56.999 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:35:56.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:35:57.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:35:57.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:35:57.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:35:58.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:35:58.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:35:58.275 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:35:58.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:35:58.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:35:58.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:35:58.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:35:59.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:35:59.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:35:59.550 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:35:59.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:35:59.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:35:59.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:35:59.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:00.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:00.825 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:36:00.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:00.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:36:00.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:00.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:36:01.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:02.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:02.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:02.101 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:36:02.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:36:02.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:02.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:36:02.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:03.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:03.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:03.379 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:36:03.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:36:03.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:03.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:36:03.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:04.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:04.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:04.703 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:36:04.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:36:04.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:04.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:36:04.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:05.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:05.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:05.975 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:36:05.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:36:05.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:05.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:36:06.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:07.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:07.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:07.253 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:36:07.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:36:07.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:07.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:36:07.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:08.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:08.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:08.527 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:36:08.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:36:08.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:08.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:36:09.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:10.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:10.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:10.193 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:36:10.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:36:10.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:10.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:36:10.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:11.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:11.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:11.959 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:36:11.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:36:11.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:11.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:36:12.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:13.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:13.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:13.730 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:36:13.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:36:13.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:13.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:36:13.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:14.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:14.962 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:36:14.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:14.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:36:14.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:14.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:36:15.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:16.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:16.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:16.481 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:36:16.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:36:16.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:16.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:36:17.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:18.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:18.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:18.041 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:36:18.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:36:18.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:18.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:36:18.310 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3855110831,v1:127.0.0.1:6803/3855110831] [v2:127.0.0.1:6804/3855110831,v1:127.0.0.1:6805/3855110831] exists,up 71f6b0bb-765a-46a1-b527-b89fce595765 2024-08-19T14:36:18.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:36:18.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:36:18.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:36:18.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-08-19T14:36:18.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:36:18.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:36:18.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:36:18.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T14:36:18.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:36:18.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:36:18.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:36:18.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:36:18.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:36:18.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:36:18.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:36:18.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:36:18.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:36:18.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:36:18.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:36:18.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:36:18.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:36:18.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:36:18.316 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:36:18.317 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:36:18.317 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:36:18.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:36:18.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:36:18.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:36:18.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:36:18.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:36:18.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:36:18.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:36:18.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:36:18.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:36:18.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:36:18.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:36:18.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:36:18.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:36:18.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:36:18.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:36:18.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=db64104a-a23f-487a-86aa-36ef598b5e5b 2024-08-19T14:36:18.318 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 db64104a-a23f-487a-86aa-36ef598b5e5b 2024-08-19T14:36:18.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 db64104a-a23f-487a-86aa-36ef598b5e5b' 2024-08-19T14:36:18.319 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:36:18.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBiWMNmqO3KExAAYE7XgS5R2+eyNDhLe10YOA== 2024-08-19T14:36:18.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBiWMNmqO3KExAAYE7XgS5R2+eyNDhLe10YOA=="}' 2024-08-19T14:36:18.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new db64104a-a23f-487a-86aa-36ef598b5e5b -i td/osd-scrub-repair/1/new.json 2024-08-19T14:36:18.650 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:36:18.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T14:36:18.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBiWMNmqO3KExAAYE7XgS5R2+eyNDhLe10YOA== --osd-uuid db64104a-a23f-487a-86aa-36ef598b5e5b 2024-08-19T14:36:18.693 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:18.692+0000 7f5c2e063640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:36:18.793 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:18.792+0000 7f5c2e063640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:36:18.794 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:18.793+0000 7f5c2e063640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:36:18.795 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:18.794+0000 7f5c2e063640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:36:18.795 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:18.794+0000 7f5c2e063640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:36:18.795 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:18.794+0000 7f5c2e063640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:36:18.871 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:18.870+0000 7f5c2e063640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T14:36:24.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T14:36:24.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:36:24.808 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T14:36:24.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T14:36:24.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:36:25.154 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T14:36:25.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T14:36:25.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:36:25.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:36:25.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:36:25.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:36:25.185 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:25.184+0000 7f2bde010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:36:25.186 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:25.186+0000 7f2bde010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:36:25.188 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:25.187+0000 7f2bde010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:36:25.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T14:36:25.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:36:25.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:36:25.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:36:25.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:36:25.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:25.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:36:25.422 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:36:25.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:25.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:25.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:25.715 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:25.715+0000 7f2bde010640 -1 Falling back to public interface 2024-08-19T14:36:26.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:26.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:26.693 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:36:26.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:36:26.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:26.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:26.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:27.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:27.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:27.972 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:36:27.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:36:27.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:27.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:28.046 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:28.045+0000 7f2bde010640 -1 osd.1 0 log_to_monitors true 2024-08-19T14:36:28.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:29.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:29.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:29.248 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:36:29.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:36:29.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:29.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:29.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:30.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:30.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:30.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:36:30.539 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:36:30.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:30.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:30.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:31.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:31.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:31.963 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:36:31.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:36:31.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:31.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:32.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:33.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:33.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:33.238 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:36:33.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:36:33.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:33.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:33.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:34.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:34.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:34.614 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:36:34.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:36:34.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:34.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:34.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:35.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:35.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:35.965 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:36:35.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:36:35.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:35.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:36.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:37.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:37.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:37.245 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:36:37.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:36:37.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:37.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:37.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:38.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:38.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:38.516 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:36:38.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:36:38.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:38.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:38.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:39.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:39.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:39.969 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:36:39.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:36:39.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:39.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:40.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:41.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:41.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:41.248 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:36:41.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:36:41.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:41.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:41.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:42.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:42.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:42.530 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:36:42.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:36:42.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:42.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:42.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:43.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:43.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:43.987 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:36:43.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:36:43.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:43.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:44.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:45.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:45.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:45.567 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:36:45.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:36:45.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:45.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:46.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:47.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:47.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:47.028 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:36:47.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:36:47.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:47.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:47.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:48.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:48.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:48.629 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:36:48.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:36:48.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:48.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:48.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:49.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:49.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:49.980 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:36:49.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:36:49.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:49.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:50.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:36:51.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:36:51.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:51.727 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T14:36:51.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T14:36:51.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:51.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:36:51.994 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1915630796,v1:127.0.0.1:6811/1915630796] [v2:127.0.0.1:6812/1915630796,v1:127.0.0.1:6813/1915630796] exists,up db64104a-a23f-487a-86aa-36ef598b5e5b 2024-08-19T14:36:51.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:36:51.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:36:51.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:36:51.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-08-19T14:36:51.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:36:51.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:36:51.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:36:51.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T14:36:51.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:36:51.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T14:36:51.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:36:51.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:36:51.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:36:51.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:36:51.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T14:36:51.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T14:36:51.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:36:51.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:36:52.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:36:52.000 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:36:52.000 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:36:52.000 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:36:52.000 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:36:52.000 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:36:52.000 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:36:52.000 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:36:52.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:36:52.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:36:52.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:36:52.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:36:52.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:36:52.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:36:52.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:36:52.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:36:52.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:36:52.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:36:52.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:36:52.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:36:52.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T14:36:52.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:36:52.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9a10a35f-6c1e-4792-bc97-cb32dcb703d9 2024-08-19T14:36:52.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9a10a35f-6c1e-4792-bc97-cb32dcb703d9' 2024-08-19T14:36:52.002 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 9a10a35f-6c1e-4792-bc97-cb32dcb703d9 2024-08-19T14:36:52.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:36:52.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCEWMNmdAPoABAAaI6ZS2uDxfFJVMesmuMJGA== 2024-08-19T14:36:52.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCEWMNmdAPoABAAaI6ZS2uDxfFJVMesmuMJGA=="}' 2024-08-19T14:36:52.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9a10a35f-6c1e-4792-bc97-cb32dcb703d9 -i td/osd-scrub-repair/2/new.json 2024-08-19T14:36:52.403 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:36:52.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-08-19T14:36:52.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCEWMNmdAPoABAAaI6ZS2uDxfFJVMesmuMJGA== --osd-uuid 9a10a35f-6c1e-4792-bc97-cb32dcb703d9 2024-08-19T14:36:52.446 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:52.444+0000 7f8e4aa19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:36:52.471 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:52.469+0000 7f8e4aa19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:36:52.472 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:52.470+0000 7f8e4aa19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:36:52.472 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:52.471+0000 7f8e4aa19640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:36:52.472 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:52.471+0000 7f8e4aa19640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:36:52.473 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:52.471+0000 7f8e4aa19640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:36:52.537 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:52.536+0000 7f8e4aa19640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-08-19T14:36:58.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-08-19T14:36:58.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:36:58.444 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T14:36:58.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T14:36:58.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:36:58.794 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T14:36:58.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T14:36:58.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:36:58.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:36:58.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:36:58.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:36:58.824 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:58.824+0000 7f733d121640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:36:58.835 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:58.834+0000 7f733d121640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:36:58.836 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:58.835+0000 7f733d121640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:36:59.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T14:36:59.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:36:59.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T14:36:59.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:36:59.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:36:59.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:36:59.096 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:36:59.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:36:59.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:36:59.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:36:59.365 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:36:59.365+0000 7f733d121640 -1 Falling back to public interface 2024-08-19T14:36:59.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:00.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:00.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:00.405 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:37:00.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:37:00.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:00.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:37:00.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:01.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:01.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:01.699 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:37:01.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:37:01.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:01.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:37:01.895 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:37:01.895+0000 7f733d121640 -1 osd.2 0 log_to_monitors true 2024-08-19T14:37:02.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:03.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:03.022 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:37:03.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:03.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:37:03.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:03.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:37:03.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:04.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:04.313 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:37:04.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:04.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:37:04.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:04.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:37:04.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:05.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:05.591 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:37:05.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:05.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:37:05.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:05.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:37:05.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:06.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:06.963 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:37:06.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:06.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:37:06.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:06.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:37:07.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:08.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:08.241 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:37:08.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:08.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:37:08.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:08.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:37:08.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:09.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:09.580 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:37:09.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:09.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:37:09.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:09.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:37:09.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:10.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:10.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:10.887 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:37:10.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:37:10.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:10.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:37:11.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:12.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:12.177 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:37:12.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:12.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:37:12.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:12.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:37:12.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:13.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:13.454 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:37:13.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:13.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:37:13.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:13.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:37:14.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:15.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:15.271 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:37:15.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:15.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:37:15.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:15.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:37:16.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:17.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:17.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:17.091 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:37:17.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:37:17.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:17.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:37:17.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:18.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:18.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:18.795 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:37:18.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:37:18.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:18.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:37:19.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:20.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:20.573 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:37:20.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:20.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:37:20.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:20.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:37:20.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:21.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:21.854 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:37:21.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:21.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:37:21.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:21.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:37:22.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:23.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:23.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:23.262 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:37:23.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:37:23.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:23.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:37:24.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:25.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:25.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:25.157 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:37:25.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:37:25.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:25.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:37:25.433 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/323754081,v1:127.0.0.1:6819/323754081] [v2:127.0.0.1:6820/323754081,v1:127.0.0.1:6821/323754081] exists,up 9a10a35f-6c1e-4792-bc97-cb32dcb703d9 2024-08-19T14:37:25.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:37:25.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:37:25.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:37:25.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-08-19T14:37:25.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T14:37:25.711 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-08-19T14:37:25.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T14:37:25.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T14:37:26.076 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' created 2024-08-19T14:37:26.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T14:37:27.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T14:37:29.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-08-19T14:37:29.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:37:29.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:37:29.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:37:29.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:37:29.118 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:37:29.118 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:37:29.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:37:29.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:37:29.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:37:29.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:37:29.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:37:29.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:37:29.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:37:29.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:37:29.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:37:29.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:37:29.525 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:37:29.525 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:37:29.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:37:29.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:37:29.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:37:29.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-08-19T14:37:29.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-08-19T14:37:29.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-08-19T14:37:29.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:37:29.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:37:29.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476750 2024-08-19T14:37:29.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476750 2024-08-19T14:37:29.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-68719476750' 2024-08-19T14:37:29.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:37:29.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:37:29.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953477 2024-08-19T14:37:29.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953477 2024-08-19T14:37:29.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-68719476750 2-137438953477' 2024-08-19T14:37:29.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:37:29.885 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-08-19T14:37:29.885 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:37:29.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:37:29.886 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-08-19T14:37:29.886 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:37:29.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-08-19T14:37:29.887 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836498 2024-08-19T14:37:29.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-08-19T14:37:29.887 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:37:30.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-08-19T14:37:30.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:37:31.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:37:31.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:37:31.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-08-19T14:37:31.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:37:31.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476750 2024-08-19T14:37:31.433 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:37:31.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:37:31.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476750 2024-08-19T14:37:31.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:37:31.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476750 2024-08-19T14:37:31.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476750' 2024-08-19T14:37:31.435 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 68719476750 2024-08-19T14:37:31.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:37:31.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476750 -lt 68719476750 2024-08-19T14:37:31.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:37:31.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953477 2024-08-19T14:37:31.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:37:31.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:37:31.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953477 2024-08-19T14:37:31.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:37:31.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953477 2024-08-19T14:37:31.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953477' 2024-08-19T14:37:31.708 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 137438953477 2024-08-19T14:37:31.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:37:31.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953478 -lt 137438953477 2024-08-19T14:37:31.974 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:37:31.974 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:37:31.974 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:37:32.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T14:37:32.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:37:32.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:37:32.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:37:32.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:37:32.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:37:32.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:37:32.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:37:32.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:37:32.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:37:32.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:37:32.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:37:32.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T14:37:32.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:37:32.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:37:32.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-08-19T14:37:32.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-08-19T14:37:32.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-08-19T14:37:32.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-08-19T14:37:32.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-08-19T14:37:32.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-08-19T14:37:33.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-08-19T14:37:33.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-08-19T14:37:34.110 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'ecpool' created 2024-08-19T14:37:34.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T14:37:35.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-08-19T14:37:35.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-08-19T14:37:35.544 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 2 allow_ec_overwrites to true 2024-08-19T14:37:35.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-08-19T14:37:35.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:37:35.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:37:35.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:37:35.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:37:35.557 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:37:35.557 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:37:35.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:37:35.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:37:35.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:37:35.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:37:35.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:37:35.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:37:35.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:37:35.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:37:35.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:37:36.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:37:36.042 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:37:36.042 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:37:36.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:37:36.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:37:36.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:37:36.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-08-19T14:37:36.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-08-19T14:37:36.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-08-19T14:37:36.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:37:36.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:37:36.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476752 2024-08-19T14:37:36.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476752 2024-08-19T14:37:36.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-68719476752' 2024-08-19T14:37:36.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:37:36.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:37:36.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953480 2024-08-19T14:37:36.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953480 2024-08-19T14:37:36.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-68719476752 2-137438953480' 2024-08-19T14:37:36.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:37:36.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-08-19T14:37:36.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:37:36.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:37:36.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-08-19T14:37:36.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:37:36.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-08-19T14:37:36.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-08-19T14:37:36.354 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836500 2024-08-19T14:37:36.354 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:37:36.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-08-19T14:37:36.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:37:37.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:37:37.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:37:37.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-08-19T14:37:37.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:37:37.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476752 2024-08-19T14:37:37.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:37:37.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:37:37.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476752 2024-08-19T14:37:37.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:37:37.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476752 2024-08-19T14:37:37.903 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 68719476752 2024-08-19T14:37:37.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476752' 2024-08-19T14:37:37.903 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:37:38.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476753 -lt 68719476752 2024-08-19T14:37:38.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:37:38.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953480 2024-08-19T14:37:38.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:37:38.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:37:38.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953480 2024-08-19T14:37:38.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:37:38.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953480 2024-08-19T14:37:38.173 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 137438953480 2024-08-19T14:37:38.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953480' 2024-08-19T14:37:38.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:37:38.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953480 -lt 137438953480 2024-08-19T14:37:38.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:37:38.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:37:38.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:37:38.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:37:38.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:37:38.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:37:38.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:37:38.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:37:38.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:37:38.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:37:38.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:37:39.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:37:39.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:37:39.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:37:39.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:37:39.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:37:39.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:37:39.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:37:39.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-08-19T14:37:39.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-08-19T14:37:39.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-08-19T14:37:39.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-08-19T14:37:39.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-08-19T14:37:39.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-08-19T14:37:39.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-08-19T14:37:39.563 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-08-19T14:37:39.563 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-08-19T14:37:39.563 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:37:39.563 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:37:39.563 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:37:39.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-08-19T14:37:39.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:37:39.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-08-19T14:37:40.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:37:40.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:37:40.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:37:40.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T14:37:40.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:37:40.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:37:40.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:37:40.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:37:40.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T14:37:40.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:37:40.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T14:37:40.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:37:40.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:37:40.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:37:40.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:37:40.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:37:40.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:37:40.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:37:40.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:37:40.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:37:40.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T14:37:40.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:37:40.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:37:40.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-08-19T14:37:42.396 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-08-19T14:37:42.396 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-08-19T14:37:42.396 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-08-19T14:37:42.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:37:42.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:37:42.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:37:42.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:37:42.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:37:42.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:37:42.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:37:42.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:37:42.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:37:42.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:37:42.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:37:42.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:37:42.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:37:42.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:37:42.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:37:42.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:37:42.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:37:42.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:37:42.914 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:37:42.914 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:37:42.914 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:37:42.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:37:42.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:37:42.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:37:42.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:37:42.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:37:42.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:37:42.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:37:42.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:37:42.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:37:42.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:37:42.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:37:42.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:37:42.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:37:42.916 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T14:37:42.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:37:42.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T14:37:42.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:37:42.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:37:42.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:37:42.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:37:42.946 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:37:42.945+0000 7f4254a20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:37:42.948 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:37:42.947+0000 7f4254a20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:37:42.950 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:37:42.948+0000 7f4254a20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:37:43.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:37:43.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:37:43.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:37:43.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:37:43.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:37:43.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:43.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:37:43.193 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:37:43.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:43.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:37:43.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:43.472 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:37:43.471+0000 7f4254a20640 -1 Falling back to public interface 2024-08-19T14:37:44.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:44.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:44.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:37:44.458 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:37:44.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:44.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:37:44.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:45.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:45.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:45.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:37:45.735 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:37:45.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:45.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:37:45.836 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:37:45.834+0000 7f4254a20640 -1 osd.1 51 log_to_monitors true 2024-08-19T14:37:46.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:37:47.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:37:47.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:37:47.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:37:47.049 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:37:47.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:37:47.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:37:47.349 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 52 last_clean_interval [16,51) [v2:127.0.0.1:6810/624848373,v1:127.0.0.1:6811/624848373] [v2:127.0.0.1:6812/624848373,v1:127.0.0.1:6813/624848373] exists,up db64104a-a23f-487a-86aa-36ef598b5e5b 2024-08-19T14:37:47.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:37:47.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:37:47.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:37:47.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:37:47.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:37:47.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:37:47.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:37:47.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:37:47.349 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:37:47.349 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:37:47.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:37:47.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:37:47.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:37:47.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:37:47.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:37:47.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:37:47.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:37:47.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:37:47.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:37:47.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:37:47.704 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:37:47.704 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:37:47.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:37:47.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:37:47.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:37:47.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-08-19T14:37:47.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-08-19T14:37:47.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-08-19T14:37:47.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:37:47.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:37:47.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135874 2024-08-19T14:37:47.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135874 2024-08-19T14:37:47.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-244813135874' 2024-08-19T14:37:47.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:37:47.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:37:48.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953483 2024-08-19T14:37:48.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953483 2024-08-19T14:37:48.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-244813135874 2-137438953483' 2024-08-19T14:37:48.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:37:48.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-08-19T14:37:48.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:37:48.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:37:48.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-08-19T14:37:48.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:37:48.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-08-19T14:37:48.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-08-19T14:37:48.073 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836503 2024-08-19T14:37:48.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:37:48.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-08-19T14:37:48.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:37:49.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:37:49.339 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:37:49.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836503 2024-08-19T14:37:49.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:37:49.607 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135874 2024-08-19T14:37:49.607 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:37:49.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:37:49.608 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135874 2024-08-19T14:37:49.608 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:37:49.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135874 2024-08-19T14:37:49.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-08-19T14:37:49.609 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 244813135874 2024-08-19T14:37:49.610 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:37:49.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135874 -lt 244813135874 2024-08-19T14:37:49.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:37:49.897 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953483 2024-08-19T14:37:49.897 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:37:49.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:37:49.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953483 2024-08-19T14:37:49.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:37:49.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953483 2024-08-19T14:37:49.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953483' 2024-08-19T14:37:49.900 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 137438953483 2024-08-19T14:37:49.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:37:50.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953483 -lt 137438953483 2024-08-19T14:37:50.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:37:50.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:37:50.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:37:50.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:37:50.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:37:50.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:37:50.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:37:50.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:37:50.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:37:50.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:37:50.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:37:50.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:37:50.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:37:50.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:37:50.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:37:51.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:37:51.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:37:51.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:37:51.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-08-19T14:37:51.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T14:37:51.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T14:37:51.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:37:51.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:37:51.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-08-19T14:37:51.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-08-19T14:37:51.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:37:51.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:37:51.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:37:51.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:37:51.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-08-19T14:37:49.204750+0000 2024-08-19T14:37:51.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T14:37:51.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:37:49.204750+0000 2024-08-19T14:37:51.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:37:51.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:37:51.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:37:51.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:37:51.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:37:51.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:37:51.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:37:51.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:37:51.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:37:49.204750+0000 '>' 2024-08-19T14:37:49.204750+0000 2024-08-19T14:37:51.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:37:52.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:37:52.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:37:52.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:37:52.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:37:52.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:37:52.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:37:52.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:37:53.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:37:49.204750+0000 '>' 2024-08-19T14:37:49.204750+0000 2024-08-19T14:37:53.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:37:54.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:37:54.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:37:54.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:37:54.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:37:54.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:37:54.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:37:54.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:37:54.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:37:49.204750+0000 '>' 2024-08-19T14:37:49.204750+0000 2024-08-19T14:37:54.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:37:55.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:37:55.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:37:55.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:37:55.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:37:55.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:37:55.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:37:55.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:37:55.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:37:49.204750+0000 '>' 2024-08-19T14:37:49.204750+0000 2024-08-19T14:37:55.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:37:56.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:37:56.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:37:56.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:37:56.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:37:56.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:37:56.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:37:56.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:37:57.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:37:49.204750+0000 '>' 2024-08-19T14:37:49.204750+0000 2024-08-19T14:37:57.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:37:58.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:37:58.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:37:58.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:37:58.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:37:58.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:37:58.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:37:58.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:37:58.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:37:49.204750+0000 '>' 2024-08-19T14:37:49.204750+0000 2024-08-19T14:37:58.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:37:59.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:37:59.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:37:59.377 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:37:59.377 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:37:59.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:37:59.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:37:59.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:37:59.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:37:49.204750+0000 '>' 2024-08-19T14:37:49.204750+0000 2024-08-19T14:37:59.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:38:00.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:38:00.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:38:00.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:38:00.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:38:00.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:38:00.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:38:00.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:38:00.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:37:56.176519+0000 '>' 2024-08-19T14:37:49.204750+0000 2024-08-19T14:38:00.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:38:00.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-08-19T14:38:00.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:38:00.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:38:00.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:38:00.920 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:38:00.921 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:38:00.921 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:38:00.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:38:00.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:38:00.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:38:01.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:38:01.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:38:01.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:38:01.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:38:01.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:38:01.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:38:01.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:38:01.303 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:38:01.303 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:38:01.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:38:01.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:38:01.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:38:01.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-08-19T14:38:01.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-08-19T14:38:01.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-08-19T14:38:01.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:38:01.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:38:01.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135878 2024-08-19T14:38:01.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135878 2024-08-19T14:38:01.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-244813135878' 2024-08-19T14:38:01.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:38:01.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:38:01.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953487 2024-08-19T14:38:01.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953487 2024-08-19T14:38:01.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-244813135878 2-137438953487' 2024-08-19T14:38:01.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:38:01.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-08-19T14:38:01.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:38:01.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:38:01.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-08-19T14:38:01.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:38:01.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-08-19T14:38:01.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-08-19T14:38:01.614 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836507 2024-08-19T14:38:01.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:38:01.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-08-19T14:38:01.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:38:02.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:38:02.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:38:03.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836507 2024-08-19T14:38:03.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:38:03.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135878 2024-08-19T14:38:03.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:38:03.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:38:03.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135878 2024-08-19T14:38:03.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:38:03.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135878 2024-08-19T14:38:03.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135878' 2024-08-19T14:38:03.170 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 244813135878 2024-08-19T14:38:03.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:38:03.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135878 -lt 244813135878 2024-08-19T14:38:03.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:38:03.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953487 2024-08-19T14:38:03.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:38:03.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:38:03.466 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953487 2024-08-19T14:38:03.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:38:03.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953487 2024-08-19T14:38:03.468 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 137438953487 2024-08-19T14:38:03.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953487' 2024-08-19T14:38:03.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:38:03.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953487 -lt 137438953487 2024-08-19T14:38:03.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:38:03.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:38:03.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:38:04.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:38:04.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:38:04.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:38:04.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:38:04.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:38:04.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:38:04.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:38:04.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:38:04.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:38:04.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:38:04.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:38:04.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:38:04.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:38:04.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:38:04.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:38:04.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-08-19T14:38:04.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-08-19T14:38:04.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-08-19T14:38:04.854 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-08-19T14:38:04.854 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-08-19T14:38:04.854 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:38:04.854 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:38:04.854 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:38:05.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-08-19T14:38:05.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:38:05.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-08-19T14:38:05.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:38:05.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:38:05.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:38:05.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T14:38:05.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:38:05.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:38:05.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:38:05.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:38:05.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T14:38:05.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:38:05.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T14:38:05.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:38:05.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:38:05.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:38:05.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:38:05.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:38:05.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:38:05.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:38:05.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:38:05.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:38:05.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T14:38:05.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:38:05.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:38:05.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-08-19T14:38:07.474 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-08-19T14:38:07.474 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-08-19T14:38:07.474 INFO:tasks.workunit.client.0.smithi102.stdout:_ 2024-08-19T14:38:07.474 INFO:tasks.workunit.client.0.smithi102.stdout:hinfo_key 2024-08-19T14:38:07.474 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 2024-08-19T14:38:07.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:38:07.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:38:07.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:38:07.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:38:07.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:38:07.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:38:07.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:38:07.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:38:07.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:38:07.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:38:07.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:38:07.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:38:07.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:38:07.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:38:07.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:38:07.742 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:38:07.742 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:38:07.742 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:38:07.742 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:38:07.742 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:38:07.742 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:38:07.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:38:07.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:38:07.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:38:07.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:38:07.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:38:07.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:38:07.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:38:07.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:38:07.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:38:07.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:38:07.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T14:38:07.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:38:07.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:38:07.744 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T14:38:07.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-08-19T14:38:07.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T14:38:07.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:38:07.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:38:07.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:38:07.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:38:07.775 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:07.774+0000 7fd87e20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:38:07.778 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:07.777+0000 7fd87e20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:38:07.780 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:07.779+0000 7fd87e20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:38:08.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:38:08.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:38:08.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:38:08.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:38:08.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:38:08.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:08.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:38:08.025 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:38:08.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:08.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:38:08.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:08.301 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:08.301+0000 7fd87e20f640 -1 Falling back to public interface 2024-08-19T14:38:09.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:09.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:09.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:38:09.291 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:38:09.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:09.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:38:09.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:10.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:10.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:10.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:38:10.564 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:38:10.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:10.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:38:10.566 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:10.565+0000 7fd87e20f640 -1 osd.1 69 log_to_monitors true 2024-08-19T14:38:10.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:11.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:11.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:11.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:38:11.905 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:38:11.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:11.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:38:12.278 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 76 up_thru 76 down_at 70 last_clean_interval [57,69) [v2:127.0.0.1:6810/865424734,v1:127.0.0.1:6811/865424734] [v2:127.0.0.1:6812/865424734,v1:127.0.0.1:6813/865424734] exists,up db64104a-a23f-487a-86aa-36ef598b5e5b 2024-08-19T14:38:12.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:38:12.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:38:12.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:38:12.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:38:12.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:38:12.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:38:12.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:38:12.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:38:12.279 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:38:12.279 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:38:12.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:38:12.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:38:12.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:38:12.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:38:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:38:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:38:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:38:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:38:12.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:38:12.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:38:12.674 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:38:12.674 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T14:38:12.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:38:12.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:38:12.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:38:12.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836510 2024-08-19T14:38:12.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836510 2024-08-19T14:38:12.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510' 2024-08-19T14:38:12.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:38:12.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:38:12.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514498 2024-08-19T14:38:12.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514498 2024-08-19T14:38:12.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-326417514498' 2024-08-19T14:38:12.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:38:12.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:38:12.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953490 2024-08-19T14:38:12.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953490 2024-08-19T14:38:12.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-326417514498 2-137438953490' 2024-08-19T14:38:12.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:38:12.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836510 2024-08-19T14:38:12.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:38:12.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:38:12.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836510 2024-08-19T14:38:12.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:38:12.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836510 2024-08-19T14:38:12.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-08-19T14:38:12.987 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836510 2024-08-19T14:38:12.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:38:13.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836510 2024-08-19T14:38:13.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:38:14.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:38:14.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:38:14.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836510 2024-08-19T14:38:14.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:38:15.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T14:38:15.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:38:15.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836510 2024-08-19T14:38:15.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:38:15.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-326417514498 2024-08-19T14:38:15.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:38:15.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:38:15.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-326417514498 2024-08-19T14:38:15.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:38:15.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514498 2024-08-19T14:38:15.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 326417514498' 2024-08-19T14:38:15.800 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 326417514498 2024-08-19T14:38:15.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:38:16.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514498 -lt 326417514498 2024-08-19T14:38:16.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:38:16.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953490 2024-08-19T14:38:16.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:38:16.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:38:16.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953490 2024-08-19T14:38:16.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:38:16.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953490 2024-08-19T14:38:16.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953490' 2024-08-19T14:38:16.108 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 137438953490 2024-08-19T14:38:16.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:38:16.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953490 -lt 137438953490 2024-08-19T14:38:16.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:38:16.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:38:16.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:38:16.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:38:16.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:38:16.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:38:16.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:38:16.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:38:16.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:38:16.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:38:16.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:38:17.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:38:17.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:38:17.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:38:17.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:38:17.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:38:17.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:38:17.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:38:17.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T14:38:17.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T14:38:17.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T14:38:17.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:38:17.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:38:17.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:38:17.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:38:17.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:38:17.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:38:17.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:38:17.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:38:17.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:38:17.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:38:17.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:38:17.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:38:17.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:38:17.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:38:17.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:38:17.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:38:17.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:38:17.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:38:17.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:38:17.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:38:17.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:38:17.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:38:17.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:38:17.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:38:17.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:38:17.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:38:17.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:38:17.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:38:17.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T14:38:17.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T14:38:17.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T14:38:17.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T14:38:17.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:38:17.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:38:17.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:38:17.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:38:17.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:38:17.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:38:17.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:38:17.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:38:17.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:38:17.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:38:17.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:38:17.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:38:17.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:38:17.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:38:17.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:38:17.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:38:17.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:38:17.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:38:17.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:38:17.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:38:17.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:38:17.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:38:17.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:38:17.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:38:17.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:38:17.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:38:17.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:38:17.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:38:17.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T14:38:17.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T14:38:17.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:38:17.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:38:17.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T14:38:17.552 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T14:38:17.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T14:38:17.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T14:38:17.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T14:38:17.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2024-08-19T14:38:17.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2024-08-19T14:38:17.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-08-19T14:38:17.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=false 2024-08-19T14:38:17.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-08-19T14:38:17.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-08-19T14:38:17.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T14:38:17.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T14:38:17.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T14:38:17.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T14:38:17.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T14:38:17.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-08-19T14:38:17.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T14:38:17.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:38:17.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:38:17.912 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:38:17.912 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:38:17.912 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:38:17.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:38:17.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T14:38:18.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T14:38:18.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T14:38:18.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:38:18.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:38:18.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T14:38:18.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T14:38:18.105 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:38:18.106 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:38:18.106 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:38:18.106 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:38:18.106 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:38:18.106 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:38:18.106 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:38:18.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:38:18.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T14:38:18.186 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T14:38:18.186 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:38:18.186 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:38:18.186 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T14:38:18.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T14:38:18.187 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:38:18.187 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:38:18.187 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:38:18.188 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:38:18.188 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:38:18.188 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:38:18.188 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:38:18.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:38:18.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T14:38:18.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-08-19T14:38:18.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T14:38:18.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T14:38:18.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T14:38:18.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T14:38:18.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T14:38:18.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T14:38:19.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T14:38:19.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:38:19.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:38:19.097 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:38:19.098 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:38:19.098 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:38:19.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:38:19.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:38:19.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:38:19.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-08-19T14:38:19.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-08-19T14:38:19.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-08-19T14:38:19.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:38:19.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:38:19.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T14:38:19.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:38:19.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:38:19.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:38:19.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:38:19.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:38:19.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:38:19.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:38:19.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:38:19.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:38:19.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:38:19.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:38:19.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:38:19.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:38:19.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:38:19.132 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:38:19.132 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:38:19.132 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:38:19.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:38:19.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:38:19.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:38:19.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:38:19.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:38:19.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:38:19.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:38:19.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:38:19.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:38:19.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:38:19.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:38:19.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:38:19.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:38:19.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:38:19.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:38:19.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2b2c6422-22e1-4e28-a2a8-1dd1b5604026 2024-08-19T14:38:19.134 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 2b2c6422-22e1-4e28-a2a8-1dd1b5604026 2024-08-19T14:38:19.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2b2c6422-22e1-4e28-a2a8-1dd1b5604026' 2024-08-19T14:38:19.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:38:19.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDbWMNm9JrpCBAAHyJDbMtKPYmvjoWtHDhnlw== 2024-08-19T14:38:19.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDbWMNm9JrpCBAAHyJDbMtKPYmvjoWtHDhnlw=="}' 2024-08-19T14:38:19.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2b2c6422-22e1-4e28-a2a8-1dd1b5604026 -i td/osd-scrub-repair/0/new.json 2024-08-19T14:38:19.353 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:38:19.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T14:38:19.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbWMNm9JrpCBAAHyJDbMtKPYmvjoWtHDhnlw== --osd-uuid 2b2c6422-22e1-4e28-a2a8-1dd1b5604026 2024-08-19T14:38:19.392 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:19.392+0000 7f0cc0214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:38:19.404 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:19.403+0000 7f0cc0214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:38:19.405 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:19.405+0000 7f0cc0214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:38:19.406 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:19.405+0000 7f0cc0214640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:38:19.406 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:19.405+0000 7f0cc0214640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:38:19.406 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:19.405+0000 7f0cc0214640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:38:19.487 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:19.486+0000 7f0cc0214640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T14:38:25.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T14:38:25.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:38:25.023 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T14:38:25.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T14:38:25.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:38:25.374 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T14:38:25.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T14:38:25.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:38:25.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:38:25.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:38:25.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:38:25.401 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:25.400+0000 7f8b8240c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:38:25.403 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:25.402+0000 7f8b8240c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:38:25.404 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:25.403+0000 7f8b8240c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:38:25.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T14:38:25.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:38:25.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:38:25.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:38:25.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:38:25.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:25.644 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:38:25.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:38:25.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:25.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:38:25.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:25.927 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:25.927+0000 7f8b8240c640 -1 Falling back to public interface 2024-08-19T14:38:26.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:26.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:26.909 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:38:26.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:38:26.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:26.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:38:27.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:28.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:28.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:28.183 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:38:28.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:38:28.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:28.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:38:28.228 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:28.227+0000 7f8b8240c640 -1 osd.0 0 log_to_monitors true 2024-08-19T14:38:28.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:29.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:29.512 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:38:29.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:29.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:38:29.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:29.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:38:29.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:30.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:30.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:30.789 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:38:30.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:38:30.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:30.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:38:31.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:32.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:32.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:32.066 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:38:32.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:38:32.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:32.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:38:32.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:33.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:33.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:33.342 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:38:33.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:38:33.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:33.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:38:33.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:34.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:34.619 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:38:34.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:34.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:38:34.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:34.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:38:34.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:35.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:35.998 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:38:35.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:35.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:38:35.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:35.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:38:36.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:37.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:37.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:37.271 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:38:37.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:38:37.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:37.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:38:37.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:38.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:38.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:38.547 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:38:38.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:38:38.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:38.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:38:38.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:39.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:39.824 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:38:39.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:39.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:38:39.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:39.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:38:40.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:41.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:41.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:41.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:38:41.413 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:38:41.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:41.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:38:42.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:43.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:43.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:43.081 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:38:43.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:38:43.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:43.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:38:43.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:44.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:44.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:44.850 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:38:44.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:38:44.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:44.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:38:45.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:46.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:46.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:46.189 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:38:46.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:38:46.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:46.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:38:46.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:47.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:47.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:47.843 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:38:47.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:38:47.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:47.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:38:48.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:49.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:49.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:49.475 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:38:49.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:38:49.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:49.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:38:49.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:50.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:50.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:50.938 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:38:50.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:38:50.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:50.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:38:51.207 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1698346599,v1:127.0.0.1:6803/1698346599] [v2:127.0.0.1:6804/1698346599,v1:127.0.0.1:6805/1698346599] exists,up 2b2c6422-22e1-4e28-a2a8-1dd1b5604026 2024-08-19T14:38:51.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:38:51.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:38:51.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:38:51.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-08-19T14:38:51.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-08-19T14:38:51.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:38:51.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:38:51.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T14:38:51.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:38:51.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:38:51.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:38:51.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:38:51.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:38:51.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:38:51.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:38:51.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:38:51.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:38:51.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:38:51.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:38:51.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:38:51.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:38:51.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:38:51.214 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:38:51.214 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:38:51.214 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:38:51.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:38:51.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:38:51.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:38:51.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:38:51.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:38:51.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:38:51.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:38:51.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:38:51.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:38:51.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:38:51.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:38:51.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:38:51.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:38:51.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:38:51.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:38:51.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=118c3f6f-c0bc-46e6-b0d9-b63cb0e35aeb 2024-08-19T14:38:51.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 118c3f6f-c0bc-46e6-b0d9-b63cb0e35aeb' 2024-08-19T14:38:51.216 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 118c3f6f-c0bc-46e6-b0d9-b63cb0e35aeb 2024-08-19T14:38:51.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:38:51.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7WMNmuN6qDRAAbafd1GpS1MO3OpgR+ZfvfQ== 2024-08-19T14:38:51.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7WMNmuN6qDRAAbafd1GpS1MO3OpgR+ZfvfQ=="}' 2024-08-19T14:38:51.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 118c3f6f-c0bc-46e6-b0d9-b63cb0e35aeb -i td/osd-scrub-repair/1/new.json 2024-08-19T14:38:51.554 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:38:51.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T14:38:51.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7WMNmuN6qDRAAbafd1GpS1MO3OpgR+ZfvfQ== --osd-uuid 118c3f6f-c0bc-46e6-b0d9-b63cb0e35aeb 2024-08-19T14:38:51.594 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:51.594+0000 7fe4b6c0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:38:51.630 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:51.630+0000 7fe4b6c0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:38:51.631 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:51.631+0000 7fe4b6c0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:38:51.632 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:51.631+0000 7fe4b6c0c640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:38:51.632 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:51.631+0000 7fe4b6c0c640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:38:51.632 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:51.631+0000 7fe4b6c0c640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:38:51.699 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:51.699+0000 7fe4b6c0c640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T14:38:57.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T14:38:57.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:38:57.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T14:38:57.348 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T14:38:57.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:38:57.693 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T14:38:57.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T14:38:57.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:38:57.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:38:57.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:38:57.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:38:57.721 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:57.720+0000 7fe01cf01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:38:57.722 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:57.722+0000 7fe01cf01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:38:57.723 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:57.723+0000 7fe01cf01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:38:57.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T14:38:57.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:38:57.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:38:57.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:38:57.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:38:57.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:57.962 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:38:57.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:38:57.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:57.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:38:58.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:38:58.251 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:38:58.251+0000 7fe01cf01640 -1 Falling back to public interface 2024-08-19T14:38:59.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:38:59.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:38:59.227 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:38:59.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:38:59.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:38:59.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:38:59.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:00.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:00.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:00.500 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:39:00.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:39:00.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:00.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:39:00.561 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:00.561+0000 7fe01cf01640 -1 osd.1 0 log_to_monitors true 2024-08-19T14:39:00.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:01.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:01.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:01.778 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:39:01.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:39:01.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:01.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:39:02.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:03.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:03.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:03.165 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:39:03.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:39:03.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:03.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:39:03.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:04.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:04.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:04.539 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:39:04.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:39:04.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:04.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:39:04.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:05.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:05.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:05.817 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:39:05.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:39:05.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:05.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:39:06.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:07.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:07.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:07.090 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:39:07.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:39:07.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:07.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:39:07.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:08.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:08.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:08.465 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:39:08.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:39:08.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:08.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:39:08.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:09.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:09.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:09.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:39:09.741 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:39:09.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:09.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:39:10.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:11.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:11.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:11.018 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:39:11.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:39:11.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:11.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:39:11.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:12.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:12.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:12.349 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:39:12.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:39:12.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:12.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:39:12.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:13.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:13.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:13.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:39:13.865 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:39:13.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:13.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:39:14.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:15.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:15.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:15.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:39:15.589 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:39:15.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:15.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:39:16.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:17.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:17.136 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:39:17.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:17.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:39:17.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:17.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:39:17.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:18.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:18.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:18.595 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:39:18.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:39:18.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:18.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:39:19.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:20.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:20.582 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:39:20.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:20.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:39:20.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:20.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:39:21.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:22.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:22.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:22.160 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:39:22.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:39:22.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:22.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:39:22.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:23.485 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:39:23.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:39:23.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:23.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:39:23.775 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1480265203,v1:127.0.0.1:6811/1480265203] [v2:127.0.0.1:6812/1480265203,v1:127.0.0.1:6813/1480265203] exists,up 118c3f6f-c0bc-46e6-b0d9-b63cb0e35aeb 2024-08-19T14:39:23.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:39:23.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:39:23.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:39:23.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-08-19T14:39:23.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-08-19T14:39:23.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:39:23.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:39:23.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T14:39:23.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:39:23.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T14:39:23.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:39:23.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:39:23.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:39:23.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:39:23.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T14:39:23.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T14:39:23.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:39:23.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:39:23.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:39:23.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:39:23.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:39:23.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:39:23.782 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:39:23.782 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:39:23.782 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:39:23.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:39:23.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:39:23.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:39:23.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:39:23.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:39:23.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:39:23.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:39:23.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:39:23.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:39:23.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:39:23.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:39:23.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:39:23.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:39:23.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T14:39:23.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:39:23.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=30952570-c55e-4456-b25b-267985f3e7cc 2024-08-19T14:39:23.783 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 30952570-c55e-4456-b25b-267985f3e7cc 2024-08-19T14:39:23.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 30952570-c55e-4456-b25b-267985f3e7cc' 2024-08-19T14:39:23.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:39:23.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAbWcNmr/16LxAAEkQZ4rHdSWQJ9fjAXDy7qA== 2024-08-19T14:39:23.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAbWcNmr/16LxAAEkQZ4rHdSWQJ9fjAXDy7qA=="}' 2024-08-19T14:39:23.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 30952570-c55e-4456-b25b-267985f3e7cc -i td/osd-scrub-repair/2/new.json 2024-08-19T14:39:24.155 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:39:24.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-08-19T14:39:24.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAbWcNmr/16LxAAEkQZ4rHdSWQJ9fjAXDy7qA== --osd-uuid 30952570-c55e-4456-b25b-267985f3e7cc 2024-08-19T14:39:24.196 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:24.194+0000 7facbe21a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:39:24.232 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:24.230+0000 7facbe21a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:39:24.233 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:24.231+0000 7facbe21a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:39:24.233 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:24.231+0000 7facbe21a640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:39:24.233 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:24.231+0000 7facbe21a640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:39:24.233 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:24.231+0000 7facbe21a640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:39:24.298 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:24.297+0000 7facbe21a640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-08-19T14:39:30.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-08-19T14:39:30.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:39:30.110 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T14:39:30.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T14:39:30.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:39:30.459 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T14:39:30.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T14:39:30.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:39:30.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:39:30.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:39:30.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:39:30.486 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:30.485+0000 7f59a4019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:39:30.488 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:30.487+0000 7f59a4019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:39:30.489 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:30.488+0000 7f59a4019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:39:30.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T14:39:30.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:39:30.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T14:39:30.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:39:30.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:39:30.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:30.723 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:39:30.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:39:30.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:30.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:30.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:31.018 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:31.018+0000 7f59a4019640 -1 Falling back to public interface 2024-08-19T14:39:31.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:31.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:31.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:39:31.990 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:39:31.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:31.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:32.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:33.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:33.264 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:39:33.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:33.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:39:33.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:33.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:33.472 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:33.472+0000 7f59a4019640 -1 osd.2 0 log_to_monitors true 2024-08-19T14:39:33.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:34.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:34.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:34.573 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:39:34.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:39:34.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:34.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:34.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:35.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:35.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:35.849 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:39:35.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:39:35.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:35.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:36.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:37.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:37.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:37.164 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:39:37.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:39:37.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:37.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:37.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:38.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:38.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:38.596 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:39:38.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:39:38.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:38.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:38.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:39.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:39.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:39.869 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:39:39.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:39:39.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:39.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:40.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:41.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:41.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:41.175 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:39:41.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:39:41.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:41.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:41.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:42.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:42.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:42.621 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:39:42.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:39:42.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:42.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:42.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:43.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:43.895 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:39:43.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:43.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:39:43.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:43.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:44.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:45.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:45.177 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:39:45.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:45.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:39:45.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:45.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:45.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:46.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:46.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:46.476 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:39:46.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:39:46.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:46.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:46.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:47.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:47.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:47.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:39:47.874 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:39:47.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:47.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:48.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:49.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:49.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:49.220 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:39:49.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:39:49.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:49.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:49.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:50.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:50.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:50.570 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:39:50.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:39:50.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:50.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:51.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:52.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:52.277 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:39:52.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:52.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:39:52.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:52.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:52.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:53.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:53.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:53.804 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:39:53.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:39:53.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:53.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:54.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:55.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:55.424 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:39:55.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:55.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:39:55.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:55.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:55.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:39:56.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:39:56.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:39:56.783 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T14:39:56.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T14:39:56.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:39:56.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:39:57.056 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3731697517,v1:127.0.0.1:6819/3731697517] [v2:127.0.0.1:6820/3731697517,v1:127.0.0.1:6821/3731697517] exists,up 30952570-c55e-4456-b25b-267985f3e7cc 2024-08-19T14:39:57.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:39:57.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:39:57.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:39:57.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-08-19T14:39:57.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-08-19T14:39:57.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:39:57.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:39:57.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-08-19T14:39:57.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:39:57.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:39:57.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:39:57.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:39:57.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:39:57.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:39:57.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T14:39:57.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T14:39:57.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:39:57.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:39:57.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:39:57.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:39:57.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:39:57.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:39:57.062 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:39:57.062 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:39:57.062 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:39:57.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:39:57.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:39:57.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:39:57.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:39:57.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:39:57.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:39:57.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:39:57.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:39:57.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:39:57.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:39:57.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:39:57.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:39:57.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:39:57.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T14:39:57.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:39:57.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=11030134-f10d-4d38-9e2b-e4fab1d556d5 2024-08-19T14:39:57.064 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 11030134-f10d-4d38-9e2b-e4fab1d556d5 2024-08-19T14:39:57.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 11030134-f10d-4d38-9e2b-e4fab1d556d5' 2024-08-19T14:39:57.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:39:57.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA9WcNmK32tBBAAuTp9/q/D9Zo+8USTBSWg4Q== 2024-08-19T14:39:57.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA9WcNmK32tBBAAuTp9/q/D9Zo+8USTBSWg4Q=="}' 2024-08-19T14:39:57.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 11030134-f10d-4d38-9e2b-e4fab1d556d5 -i td/osd-scrub-repair/3/new.json 2024-08-19T14:39:57.419 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:39:57.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-08-19T14:39:57.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA9WcNmK32tBBAAuTp9/q/D9Zo+8USTBSWg4Q== --osd-uuid 11030134-f10d-4d38-9e2b-e4fab1d556d5 2024-08-19T14:39:57.459 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:57.458+0000 7f164ca0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:39:57.499 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:57.499+0000 7f164ca0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:39:57.500 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:57.500+0000 7f164ca0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:39:57.501 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:57.500+0000 7f164ca0d640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-08-19T14:39:57.501 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:57.500+0000 7f164ca0d640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-08-19T14:39:57.501 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:57.500+0000 7f164ca0d640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-08-19T14:39:57.570 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:39:57.570+0000 7f164ca0d640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-08-19T14:40:03.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-08-19T14:40:03.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:40:03.523 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-08-19T14:40:03.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-08-19T14:40:03.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:40:03.874 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-08-19T14:40:03.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-08-19T14:40:03.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:40:03.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:40:03.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:40:03.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:40:03.902 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:40:03.901+0000 7f6c92012640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:40:03.903 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:40:03.903+0000 7f6c92012640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:40:03.904 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:40:03.904+0000 7f6c92012640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:40:04.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-08-19T14:40:04.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:40:04.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T14:40:04.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:40:04.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:40:04.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:04.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:40:04.144 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:40:04.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:04.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:04.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:04.441 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:40:04.440+0000 7f6c92012640 -1 Falling back to public interface 2024-08-19T14:40:05.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:05.410 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:40:05.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:05.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:40:05.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:05.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:05.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:06.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:06.685 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:40:06.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:06.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:40:06.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:06.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:06.730 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:40:06.729+0000 7f6c92012640 -1 osd.3 0 log_to_monitors true 2024-08-19T14:40:06.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:07.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:07.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:07.963 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:40:07.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:40:07.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:07.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:08.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:09.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:09.241 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:40:09.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:09.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:40:09.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:09.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:09.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:10.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:10.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:10.518 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:40:10.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:40:10.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:10.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:10.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:11.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:11.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:11.796 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:40:11.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:40:11.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:11.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:12.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:13.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:13.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:13.186 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:40:13.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:40:13.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:13.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:13.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:14.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:14.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:14.479 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:40:14.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:40:14.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:14.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:14.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:15.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:15.755 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:40:15.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:15.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:40:15.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:15.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:16.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:17.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:17.034 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:40:17.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:17.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:40:17.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:17.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:17.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:18.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:18.315 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:40:18.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:18.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:40:18.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:18.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:18.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:19.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:19.590 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:40:19.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:19.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:40:19.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:19.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:20.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:21.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:21.264 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:40:21.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:21.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:40:21.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:21.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:21.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:22.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:22.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:22.839 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:40:22.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:40:22.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:22.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:23.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:24.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:24.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:24.371 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:40:24.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:40:24.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:24.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:24.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:25.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:25.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:25.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:40:25.644 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:40:25.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:25.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:26.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:27.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:27.291 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:40:27.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:27.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:40:27.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:27.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:27.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:28.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:28.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:28.944 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:40:28.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:40:28.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:28.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:29.313 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/926082185,v1:127.0.0.1:6827/926082185] [v2:127.0.0.1:6828/926082185,v1:127.0.0.1:6829/926082185] exists,up 11030134-f10d-4d38-9e2b-e4fab1d556d5 2024-08-19T14:40:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:40:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:40:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:40:29.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-08-19T14:40:29.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T14:40:29.591 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-08-19T14:40:29.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T14:40:29.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T14:40:29.973 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' created 2024-08-19T14:40:29.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T14:40:30.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T14:40:33.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-08-19T14:40:33.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:40:33.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:40:33.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:40:33.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:40:33.459 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:40:33.459 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:40:33.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:40:33.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:40:33.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:40:33.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:40:33.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:40:33.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:40:33.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:40:33.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:40:33.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:40:33.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:40:33.823 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:40:33.823 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:40:33.824 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-08-19T14:40:33.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:40:33.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:40:33.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:40:33.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-08-19T14:40:33.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-08-19T14:40:33.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-08-19T14:40:33.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:40:33.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:40:34.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672984 2024-08-19T14:40:34.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672984 2024-08-19T14:40:34.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672984' 2024-08-19T14:40:34.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:40:34.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:40:34.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-08-19T14:40:34.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-08-19T14:40:34.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672984 2-64424509457' 2024-08-19T14:40:34.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:40:34.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:40:34.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378630 2024-08-19T14:40:34.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378630 2024-08-19T14:40:34.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672984 2-64424509457 3-81604378630' 2024-08-19T14:40:34.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:40:34.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-08-19T14:40:34.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:40:34.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:40:34.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-08-19T14:40:34.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:40:34.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-08-19T14:40:34.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-08-19T14:40:34.235 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836505 2024-08-19T14:40:34.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:40:34.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-08-19T14:40:34.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:40:35.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:40:35.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:40:35.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-08-19T14:40:35.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:40:35.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672984 2024-08-19T14:40:35.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:40:35.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:40:35.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672984 2024-08-19T14:40:35.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:40:35.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672984 2024-08-19T14:40:35.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-08-19T14:40:35.776 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672984 2024-08-19T14:40:35.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:40:36.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672984 2024-08-19T14:40:36.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:40:36.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-08-19T14:40:36.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:40:36.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:40:36.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-08-19T14:40:36.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:40:36.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-08-19T14:40:36.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-08-19T14:40:36.044 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509457 2024-08-19T14:40:36.045 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:40:36.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-08-19T14:40:36.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:40:36.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-81604378630 2024-08-19T14:40:36.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:40:36.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:40:36.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-81604378630 2024-08-19T14:40:36.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:40:36.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378630 2024-08-19T14:40:36.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 81604378630' 2024-08-19T14:40:36.317 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 81604378630 2024-08-19T14:40:36.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:40:36.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378630 -lt 81604378630 2024-08-19T14:40:36.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:40:36.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:40:36.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:40:36.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T14:40:36.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:40:36.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:40:36.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:40:36.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:40:36.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:40:36.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:40:36.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:40:37.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:40:37.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:40:37.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:40:37.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:40:37.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T14:40:37.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:40:37.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:40:37.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2024-08-19T14:40:37.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-08-19T14:40:37.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-08-19T14:40:37.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-08-19T14:40:37.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-08-19T14:40:37.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-08-19T14:40:37.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-08-19T14:40:37.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-08-19T14:40:38.557 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'ecpool' created 2024-08-19T14:40:38.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T14:40:39.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-08-19T14:40:39.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-08-19T14:40:39.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:40:39.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:40:39.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:40:39.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:40:39.571 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:40:39.571 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:40:39.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:40:39.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:40:39.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:40:39.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:40:39.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:40:39.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:40:39.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:40:39.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:40:39.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:40:39.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:40:39.954 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:40:39.954 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:40:39.954 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-08-19T14:40:39.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:40:39.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:40:39.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:40:40.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-08-19T14:40:40.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-08-19T14:40:40.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-08-19T14:40:40.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:40:40.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:40:40.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-08-19T14:40:40.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-08-19T14:40:40.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672986' 2024-08-19T14:40:40.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:40:40.186 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:40:40.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-08-19T14:40:40.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-08-19T14:40:40.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672986 2-64424509459' 2024-08-19T14:40:40.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:40:40.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:40:40.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378633 2024-08-19T14:40:40.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378633 2024-08-19T14:40:40.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672986 2-64424509459 3-81604378633' 2024-08-19T14:40:40.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:40:40.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-08-19T14:40:40.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:40:40.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:40:40.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-08-19T14:40:40.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:40:40.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-08-19T14:40:40.443 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836507 2024-08-19T14:40:40.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-08-19T14:40:40.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:40:40.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-08-19T14:40:40.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:40:41.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:40:41.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:40:41.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836507 2024-08-19T14:40:41.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:40:41.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-08-19T14:40:41.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:40:41.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:40:41.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-08-19T14:40:41.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:40:41.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-08-19T14:40:41.989 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672986 2024-08-19T14:40:41.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-08-19T14:40:41.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:40:42.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672986 -lt 42949672986 2024-08-19T14:40:42.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:40:42.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-08-19T14:40:42.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:40:42.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:40:42.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-08-19T14:40:42.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:40:42.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-08-19T14:40:42.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-08-19T14:40:42.261 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509459 2024-08-19T14:40:42.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:40:42.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-08-19T14:40:42.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:40:42.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-81604378633 2024-08-19T14:40:42.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:40:42.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:40:42.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-81604378633 2024-08-19T14:40:42.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:40:42.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378633 2024-08-19T14:40:42.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 81604378633' 2024-08-19T14:40:42.534 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 81604378633 2024-08-19T14:40:42.534 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:40:42.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378633 -lt 81604378633 2024-08-19T14:40:42.802 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:40:42.802 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:40:42.802 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:40:43.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:40:43.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:40:43.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:40:43.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:40:43.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:40:43.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:40:43.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:40:43.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:40:43.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:40:43.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:40:43.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:40:43.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:40:43.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:40:43.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:40:43.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:40:43.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-08-19T14:40:43.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-08-19T14:40:43.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-08-19T14:40:43.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-08-19T14:40:43.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-08-19T14:40:43.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T14:40:43.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T14:40:43.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-08-19T14:40:43.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T14:40:43.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T14:40:43.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T14:40:44.102 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T14:40:44.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T14:40:44.470 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T14:40:44.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T14:40:44.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T14:40:44.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-08-19T14:40:44.838 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-08-19T14:40:44.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-08-19T14:40:44.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:40:44.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:40:44.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:40:45.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-08-19T14:40:45.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-08-19T14:40:45.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-08-19T14:40:45.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-08-19T14:40:45.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-08-19T14:40:45.121 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:40:45.121 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-19T14:40:45.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-08-19T14:40:45.391 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:40:45.391 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:40:45.391 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-08-19T14:40:45.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-08-19T14:40:45.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-08-19T14:40:45.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-08-19T14:40:45.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-08-19T14:40:45.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-08-19T14:40:45.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-08-19T14:40:45.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-08-19T14:40:45.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-08-19T14:40:45.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-08-19T14:40:45.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:40:45.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:40:45.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:40:45.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-08-19T14:40:45.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:40:45.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:40:45.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:40:45.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:40:45.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-08-19T14:40:45.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:40:45.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-08-19T14:40:45.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:40:45.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:40:45.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:40:45.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:40:45.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:40:45.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:40:45.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:40:45.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:40:45.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:40:45.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-08-19T14:40:45.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:40:45.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:40:45.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-08-19T14:40:47.313 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-08-19T14:40:47.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-08-19T14:40:47.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:40:47.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:40:47.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-19T14:40:47.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:40:47.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:40:47.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:40:47.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:40:47.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:40:47.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:40:47.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T14:40:47.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T14:40:47.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:40:47.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:40:47.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:40:47.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:40:47.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:40:47.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:40:47.832 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:40:47.833 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:40:47.833 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:40:47.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:40:47.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:40:47.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:40:47.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:40:47.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:40:47.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:40:47.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:40:47.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:40:47.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:40:47.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:40:47.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:40:47.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T14:40:47.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-19T14:40:47.835 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.3 2024-08-19T14:40:47.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:40:47.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-08-19T14:40:47.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-19T14:40:47.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:40:47.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:40:47.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:40:47.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:40:47.860+0000 7fb56b43f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:40:47.864 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:40:47.863+0000 7fb56b43f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:40:47.865 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:40:47.864+0000 7fb56b43f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:40:48.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-19T14:40:48.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:40:48.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T14:40:48.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:40:48.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:40:48.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:48.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:40:48.112 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:40:48.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:48.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:48.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:48.389 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:40:48.389+0000 7fb56b43f640 -1 Falling back to public interface 2024-08-19T14:40:49.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:49.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:49.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:40:49.379 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:40:49.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:49.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:49.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:50.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:50.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:50.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:40:50.652 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:40:50.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:50.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:50.739 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:40:50.738+0000 7fb56b43f640 -1 osd.3 36 log_to_monitors true 2024-08-19T14:40:50.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:40:51.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:40:51.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:40:51.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:40:51.934 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:40:51.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:40:51.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:40:52.211 INFO:tasks.workunit.client.0.smithi102.stderr:osd.3 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [19,36) [v2:127.0.0.1:6826/4165777533,v1:127.0.0.1:6827/4165777533] [v2:127.0.0.1:6828/4165777533,v1:127.0.0.1:6829/4165777533] exists,up 11030134-f10d-4d38-9e2b-e4fab1d556d5 2024-08-19T14:40:52.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:40:52.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:40:52.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:40:52.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:40:52.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:40:52.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:40:52.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:40:52.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:40:52.212 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:40:52.212 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:40:52.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:40:52.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:40:52.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:40:52.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:40:52.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:40:52.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:40:52.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:40:52.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:40:52.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:40:52.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:40:52.651 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:40:52.651 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:40:52.651 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-08-19T14:40:52.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:40:52.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:40:52.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:40:52.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836511 2024-08-19T14:40:52.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836511 2024-08-19T14:40:52.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511' 2024-08-19T14:40:52.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:40:52.753 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:40:52.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672989 2024-08-19T14:40:52.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672989 2024-08-19T14:40:52.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-42949672989' 2024-08-19T14:40:52.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:40:52.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:40:52.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-08-19T14:40:52.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-08-19T14:40:52.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-42949672989 2-64424509463' 2024-08-19T14:40:52.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:40:52.959 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:40:53.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-08-19T14:40:53.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-08-19T14:40:53.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-42949672989 2-64424509463 3-171798691842' 2024-08-19T14:40:53.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:40:53.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836511 2024-08-19T14:40:53.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:40:53.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:40:53.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836511 2024-08-19T14:40:53.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:40:53.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836511 2024-08-19T14:40:53.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-08-19T14:40:53.065 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836511 2024-08-19T14:40:53.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:40:53.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836511 2024-08-19T14:40:53.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:40:53.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672989 2024-08-19T14:40:53.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:40:53.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:40:53.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672989 2024-08-19T14:40:53.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:40:53.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672989 2024-08-19T14:40:53.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-08-19T14:40:53.338 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949672989 2024-08-19T14:40:53.339 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:40:53.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672989 -lt 42949672989 2024-08-19T14:40:53.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:40:53.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-08-19T14:40:53.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:40:53.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:40:53.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-08-19T14:40:53.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:40:53.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-08-19T14:40:53.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-08-19T14:40:53.640 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509463 2024-08-19T14:40:53.640 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:40:53.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-08-19T14:40:53.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:40:53.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-171798691842 2024-08-19T14:40:53.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:40:53.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:40:53.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-171798691842 2024-08-19T14:40:53.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:40:53.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-08-19T14:40:53.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 171798691842' 2024-08-19T14:40:53.912 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.3 seq 171798691842 2024-08-19T14:40:53.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:40:54.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 171798691842 2024-08-19T14:40:54.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:40:55.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:40:55.180 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:40:55.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-08-19T14:40:55.466 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:40:55.466 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:40:55.466 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:40:55.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:40:55.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:40:55.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:40:55.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:40:55.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:40:55.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:40:55.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:40:55.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:40:56.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:40:56.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:40:56.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:40:56.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:40:56.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:40:56.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:40:56.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:40:56.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-08-19T14:40:56.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T14:40:56.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T14:40:56.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:40:56.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:40:56.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-08-19T14:40:56.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-08-19T14:40:56.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-08-19T14:40:56.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-08-19T14:40:56.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:40:56.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:40:56.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:40:56.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:40:56.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T14:40:38.415090+0000 2024-08-19T14:40:56.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-08-19T14:40:57.176 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-08-19T14:40:57.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-08-19T14:40:38.415090+0000 2024-08-19T14:40:57.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T14:40:57.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:40:38.415090+0000 2024-08-19T14:40:57.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:40:57.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:40:57.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:40:57.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:40:57.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:40:57.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:40:57.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:40:57.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:40:57.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:40:38.415090+0000 '>' 2024-08-19T14:40:38.415090+0000 2024-08-19T14:40:57.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:40:58.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:40:58.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:40:58.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:40:58.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:40:58.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:40:58.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:40:58.456 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:40:58.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:40:38.415090+0000 '>' 2024-08-19T14:40:38.415090+0000 2024-08-19T14:40:58.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:40:59.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:40:59.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:40:59.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:40:59.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:40:59.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:40:59.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:40:59.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:40:59.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:40:38.415090+0000 '>' 2024-08-19T14:40:38.415090+0000 2024-08-19T14:40:59.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:41:00.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:41:00.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:41:00.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:41:00.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:41:00.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:41:00.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:41:00.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:41:01.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:40:57.887686+0000 '>' 2024-08-19T14:40:38.415090+0000 2024-08-19T14:41:01.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:41:01.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:41:01.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:41:01.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:41:01.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-08-19T14:41:01.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:41:01.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:41:01.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:41:01.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:41:01.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-08-19T14:41:01.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:41:01.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-08-19T14:41:01.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:41:01.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:41:01.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:41:01.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:41:01.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:41:01.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:41:01.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:41:01.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:41:01.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:41:01.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-08-19T14:41:01.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:41:01.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:41:01.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-08-19T14:41:02.966 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-08-19T14:41:02.966 INFO:tasks.workunit.client.0.smithi102.stdout:_ 2024-08-19T14:41:02.966 INFO:tasks.workunit.client.0.smithi102.stdout:hinfo_key 2024-08-19T14:41:02.966 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 2024-08-19T14:41:03.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-08-19T14:41:03.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:41:03.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:41:03.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-19T14:41:03.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:41:03.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:41:03.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:41:03.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:41:03.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:41:03.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:41:03.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T14:41:03.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T14:41:03.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:41:03.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:41:03.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:41:03.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:41:03.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:41:03.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:41:03.235 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:41:03.235 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:41:03.235 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:41:03.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:41:03.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:41:03.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:41:03.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:41:03.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:41:03.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:41:03.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:41:03.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:41:03.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:41:03.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:41:03.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:41:03.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T14:41:03.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-19T14:41:03.237 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.3 2024-08-19T14:41:03.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:41:03.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-08-19T14:41:03.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-19T14:41:03.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:41:03.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:41:03.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:41:03.264 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:41:03.263+0000 7f583f416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:41:03.266 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:41:03.266+0000 7f583f416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:41:03.267 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:41:03.267+0000 7f583f416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:41:03.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-19T14:41:03.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:41:03.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T14:41:03.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:41:03.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:41:03.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:41:03.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:41:03.518 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:41:03.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:41:03.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:41:03.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:41:03.796 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:41:03.796+0000 7f583f416640 -1 Falling back to public interface 2024-08-19T14:41:04.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:41:04.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:41:04.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:41:04.786 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:41:04.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:41:04.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:41:05.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:41:06.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:41:06.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:41:06.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:41:06.061 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:41:06.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:41:06.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:41:06.170 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:41:06.169+0000 7f583f416640 -1 osd.3 42 log_to_monitors true 2024-08-19T14:41:06.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:41:07.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:41:07.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:41:07.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:41:07.380 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:41:07.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:41:07.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:41:07.797 INFO:tasks.workunit.client.0.smithi102.stderr:osd.3 up in weight 1 up_from 46 up_thru 46 down_at 43 last_clean_interval [40,42) [v2:127.0.0.1:6826/21917607,v1:127.0.0.1:6827/21917607] [v2:127.0.0.1:6828/21917607,v1:127.0.0.1:6829/21917607] exists,up 11030134-f10d-4d38-9e2b-e4fab1d556d5 2024-08-19T14:41:07.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:41:07.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:41:07.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:41:07.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:41:07.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:41:07.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:41:07.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:41:07.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:41:07.798 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:41:07.798 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:41:07.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:41:07.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:41:07.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:41:07.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:41:07.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:41:07.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:41:07.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:41:07.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:41:07.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:41:08.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:41:08.262 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:41:08.262 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:41:08.262 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-08-19T14:41:08.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:41:08.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:08.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:41:08.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836515 2024-08-19T14:41:08.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836515 2024-08-19T14:41:08.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515' 2024-08-19T14:41:08.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:08.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:41:08.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672994 2024-08-19T14:41:08.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672994 2024-08-19T14:41:08.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515 1-42949672994' 2024-08-19T14:41:08.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:08.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:41:08.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509467 2024-08-19T14:41:08.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509467 2024-08-19T14:41:08.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515 1-42949672994 2-64424509467' 2024-08-19T14:41:08.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:08.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:41:08.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495618 2024-08-19T14:41:08.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495618 2024-08-19T14:41:08.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515 1-42949672994 2-64424509467 3-197568495618' 2024-08-19T14:41:08.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:08.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836515 2024-08-19T14:41:08.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:08.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:41:08.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836515 2024-08-19T14:41:08.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:08.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836515 2024-08-19T14:41:08.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-08-19T14:41:08.675 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836515 2024-08-19T14:41:08.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:41:08.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836515 2024-08-19T14:41:08.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:41:09.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:41:09.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:41:10.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836515 -lt 21474836515 2024-08-19T14:41:10.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:10.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672994 2024-08-19T14:41:10.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:10.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:41:10.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672994 2024-08-19T14:41:10.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:10.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672994 2024-08-19T14:41:10.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2024-08-19T14:41:10.216 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949672994 2024-08-19T14:41:10.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:41:10.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672994 -lt 42949672994 2024-08-19T14:41:10.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:10.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509467 2024-08-19T14:41:10.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:10.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:41:10.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509467 2024-08-19T14:41:10.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:10.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509467 2024-08-19T14:41:10.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-08-19T14:41:10.488 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509467 2024-08-19T14:41:10.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:41:10.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509467 2024-08-19T14:41:10.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:10.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-197568495618 2024-08-19T14:41:10.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:10.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:41:10.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-197568495618 2024-08-19T14:41:10.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:10.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495618 2024-08-19T14:41:10.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 197568495618' 2024-08-19T14:41:10.759 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.3 seq 197568495618 2024-08-19T14:41:10.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:41:11.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495618 -lt 197568495618 2024-08-19T14:41:11.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:41:11.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:41:11.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:41:11.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:41:11.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:41:11.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:41:11.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:41:11.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:41:11.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:41:11.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:41:11.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:41:11.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:41:11.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:41:11.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:41:11.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:41:12.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:41:12.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:41:12.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:41:12.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T14:41:12.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T14:41:12.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-08-19T14:41:12.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-08-19T14:41:12.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-08-19T14:41:12.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-08-19T14:41:12.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:41:12.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:41:12.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:41:12.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T14:41:12.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:41:12.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:41:12.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:41:12.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:41:12.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T14:41:12.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:41:12.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T14:41:12.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:41:12.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:41:12.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:41:12.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:41:12.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:41:12.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:41:12.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:41:12.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:41:12.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:41:12.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T14:41:12.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:41:12.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:41:12.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-08-19T14:41:13.895 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-08-19T14:41:14.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T14:41:14.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:41:14.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:41:14.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:41:14.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:41:14.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:41:14.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:41:14.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:41:14.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:41:14.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:41:14.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:41:14.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:41:14.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:41:14.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:41:14.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:41:14.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:41:14.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:41:14.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:41:14.436 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:41:14.436 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:41:14.436 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:41:14.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:41:14.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:41:14.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:41:14.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:41:14.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:41:14.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:41:14.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:41:14.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:41:14.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:41:14.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:41:14.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:41:14.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:41:14.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:41:14.437 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T14:41:14.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:41:14.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T14:41:14.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:41:14.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:41:14.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:41:14.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:41:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:41:14.463+0000 7f87b281f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:41:14.467 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:41:14.466+0000 7f87b281f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:41:14.468 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:41:14.467+0000 7f87b281f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:41:14.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:41:14.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:41:14.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:41:14.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:41:14.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:41:14.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:41:14.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:41:14.711 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:41:14.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:41:14.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:41:14.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:41:14.993 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:41:14.993+0000 7f87b281f640 -1 Falling back to public interface 2024-08-19T14:41:15.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:41:15.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:41:15.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:41:15.977 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:41:15.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:41:15.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:41:16.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:41:17.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:41:17.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:41:17.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:41:17.247 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:41:17.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:41:17.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:41:17.324 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:41:17.324+0000 7f87b281f640 -1 osd.1 48 log_to_monitors true 2024-08-19T14:41:17.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:41:18.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:41:18.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:41:18.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:41:18.522 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:41:18.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:41:18.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:41:18.968 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 53 up_thru 53 down_at 49 last_clean_interval [10,48) [v2:127.0.0.1:6810/1298470678,v1:127.0.0.1:6811/1298470678] [v2:127.0.0.1:6812/1298470678,v1:127.0.0.1:6813/1298470678] exists,up 118c3f6f-c0bc-46e6-b0d9-b63cb0e35aeb 2024-08-19T14:41:18.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:41:18.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:41:18.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:41:18.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:41:18.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:41:18.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:41:18.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:41:18.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:41:18.969 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:41:18.969 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:41:18.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:41:18.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:41:18.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:41:19.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:41:19.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:41:19.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:41:19.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:41:19.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:41:19.058 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:41:19.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:41:19.523 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:41:19.523 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:41:19.523 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-08-19T14:41:19.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:41:19.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:19.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:41:19.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836518 2024-08-19T14:41:19.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836518 2024-08-19T14:41:19.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518' 2024-08-19T14:41:19.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:19.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:41:19.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266690 2024-08-19T14:41:19.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266690 2024-08-19T14:41:19.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-227633266690' 2024-08-19T14:41:19.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:19.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:41:19.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509470 2024-08-19T14:41:19.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509470 2024-08-19T14:41:19.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-227633266690 2-64424509470' 2024-08-19T14:41:19.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:19.830 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:41:19.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495621 2024-08-19T14:41:19.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495621 2024-08-19T14:41:19.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-227633266690 2-64424509470 3-197568495621' 2024-08-19T14:41:19.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:19.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836518 2024-08-19T14:41:19.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:19.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:41:19.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836518 2024-08-19T14:41:19.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:19.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836518 2024-08-19T14:41:19.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-08-19T14:41:19.935 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836518 2024-08-19T14:41:19.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:41:20.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836517 -lt 21474836518 2024-08-19T14:41:20.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:41:21.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:41:21.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:41:21.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836519 -lt 21474836518 2024-08-19T14:41:21.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:21.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-227633266690 2024-08-19T14:41:21.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:21.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:41:21.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-227633266690 2024-08-19T14:41:21.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:21.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266690 2024-08-19T14:41:21.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 227633266690' 2024-08-19T14:41:21.479 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 227633266690 2024-08-19T14:41:21.479 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:41:21.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266690 -lt 227633266690 2024-08-19T14:41:21.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:21.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509470 2024-08-19T14:41:21.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:21.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:41:21.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509470 2024-08-19T14:41:21.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:21.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509470 2024-08-19T14:41:21.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-08-19T14:41:21.748 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509470 2024-08-19T14:41:21.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:41:22.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509470 -lt 64424509470 2024-08-19T14:41:22.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:22.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-197568495621 2024-08-19T14:41:22.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:22.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:41:22.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-197568495621 2024-08-19T14:41:22.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:22.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495621 2024-08-19T14:41:22.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 197568495621' 2024-08-19T14:41:22.018 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.3 seq 197568495621 2024-08-19T14:41:22.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:41:22.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495621 -lt 197568495621 2024-08-19T14:41:22.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:41:22.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:41:22.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:41:22.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:41:22.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:41:22.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:41:22.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:41:22.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:41:22.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:41:22.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:41:22.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:41:22.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:41:22.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:41:22.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:41:22.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:41:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:41:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:41:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:41:23.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-08-19T14:41:23.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T14:41:23.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T14:41:23.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:41:23.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:41:23.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-08-19T14:41:23.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-08-19T14:41:23.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-08-19T14:41:23.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-08-19T14:41:23.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:41:23.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:41:23.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:41:23.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:41:23.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T14:40:57.887686+0000 2024-08-19T14:41:23.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-08-19T14:41:23.997 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-08-19T14:41:24.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-08-19T14:40:57.887686+0000 2024-08-19T14:41:24.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T14:41:24.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:40:57.887686+0000 2024-08-19T14:41:24.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:41:24.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:41:24.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:41:24.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:41:24.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:41:24.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:41:24.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:41:24.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:41:24.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:40:57.887686+0000 '>' 2024-08-19T14:40:57.887686+0000 2024-08-19T14:41:24.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:41:25.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:41:25.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:41:25.272 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:41:25.272 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:41:25.272 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:41:25.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:41:25.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:41:25.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:40:57.887686+0000 '>' 2024-08-19T14:40:57.887686+0000 2024-08-19T14:41:25.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:41:26.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:41:26.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:41:26.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:41:26.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:41:26.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:41:26.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:41:26.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:41:26.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:40:57.887686+0000 '>' 2024-08-19T14:40:57.887686+0000 2024-08-19T14:41:26.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:41:27.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:41:27.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:41:27.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:41:27.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:41:27.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:41:27.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:41:27.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:41:28.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:41:24.317930+0000 '>' 2024-08-19T14:40:57.887686+0000 2024-08-19T14:41:28.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:41:28.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:41:28.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:41:28.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:41:28.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-08-19T14:41:28.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:41:28.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:41:28.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:41:28.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:41:28.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-08-19T14:41:28.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:41:28.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-08-19T14:41:28.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:41:28.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:41:28.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:41:28.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:41:28.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:41:28.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:41:28.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:41:28.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:41:28.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:41:28.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-08-19T14:41:28.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:41:28.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:41:28.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-08-19T14:41:29.796 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-08-19T14:41:29.796 INFO:tasks.workunit.client.0.smithi102.stdout:_ 2024-08-19T14:41:29.796 INFO:tasks.workunit.client.0.smithi102.stdout:hinfo_key 2024-08-19T14:41:29.796 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 2024-08-19T14:41:30.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-08-19T14:41:30.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:41:30.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:41:30.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-19T14:41:30.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:41:30.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:41:30.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:41:30.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:41:30.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:41:30.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:41:30.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T14:41:30.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T14:41:30.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:41:30.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:41:30.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:41:30.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:41:30.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:41:30.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:41:30.078 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:41:30.078 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:41:30.078 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:41:30.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:41:30.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:41:30.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:41:30.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:41:30.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:41:30.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:41:30.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:41:30.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:41:30.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:41:30.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:41:30.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:41:30.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T14:41:30.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-19T14:41:30.080 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.3 2024-08-19T14:41:30.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:41:30.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-08-19T14:41:30.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-19T14:41:30.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:41:30.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:41:30.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:41:30.107 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:41:30.106+0000 7f63e24ea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:41:30.109 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:41:30.108+0000 7f63e24ea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:41:30.110 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:41:30.110+0000 7f63e24ea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:41:30.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-19T14:41:30.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:41:30.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T14:41:30.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:41:30.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:41:30.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:41:30.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:41:30.352 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:41:30.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:41:30.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:41:30.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:41:30.631 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:41:30.631+0000 7f63e24ea640 -1 Falling back to public interface 2024-08-19T14:41:31.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:41:31.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:41:31.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:41:31.618 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:41:31.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:41:31.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:41:31.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:41:32.745 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:41:32.744+0000 7f63e24ea640 -1 osd.3 55 log_to_monitors true 2024-08-19T14:41:32.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:41:32.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:41:32.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:41:32.888 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:41:32.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:41:32.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:41:33.252 INFO:tasks.workunit.client.0.smithi102.stderr:osd.3 up in weight 1 up_from 59 up_thru 53 down_at 56 last_clean_interval [46,55) [v2:127.0.0.1:6826/292556778,v1:127.0.0.1:6827/292556778] [v2:127.0.0.1:6828/292556778,v1:127.0.0.1:6829/292556778] exists,up 11030134-f10d-4d38-9e2b-e4fab1d556d5 2024-08-19T14:41:33.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:41:33.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:41:33.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:41:33.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:41:33.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:41:33.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:41:33.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:41:33.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:41:33.253 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:41:33.253 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:41:33.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:41:33.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:41:33.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:41:33.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:41:33.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:41:33.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:41:33.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:41:33.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:41:33.344 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:41:33.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:41:33.736 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:41:33.736 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:41:33.736 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-08-19T14:41:33.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:41:33.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:33.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:41:33.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836522 2024-08-19T14:41:33.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836522 2024-08-19T14:41:33.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522' 2024-08-19T14:41:33.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:33.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:41:34.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266694 2024-08-19T14:41:34.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266694 2024-08-19T14:41:34.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522 1-227633266694' 2024-08-19T14:41:34.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:34.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:41:34.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509474 2024-08-19T14:41:34.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509474 2024-08-19T14:41:34.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522 1-227633266694 2-64424509474' 2024-08-19T14:41:34.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:34.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:41:34.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070466 2024-08-19T14:41:34.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070466 2024-08-19T14:41:34.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522 1-227633266694 2-64424509474 3-253403070466' 2024-08-19T14:41:34.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:34.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836522 2024-08-19T14:41:34.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:34.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:41:34.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836522 2024-08-19T14:41:34.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:34.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836522 2024-08-19T14:41:34.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2024-08-19T14:41:34.338 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836522 2024-08-19T14:41:34.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:41:34.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836521 -lt 21474836522 2024-08-19T14:41:34.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:41:35.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:41:35.608 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:41:35.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836522 -lt 21474836522 2024-08-19T14:41:35.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:35.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-227633266694 2024-08-19T14:41:35.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:35.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:41:35.880 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-227633266694 2024-08-19T14:41:35.880 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:35.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266694 2024-08-19T14:41:35.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 227633266694' 2024-08-19T14:41:35.881 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 227633266694 2024-08-19T14:41:35.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:41:36.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266694 -lt 227633266694 2024-08-19T14:41:36.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:36.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509474 2024-08-19T14:41:36.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:36.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:41:36.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509474 2024-08-19T14:41:36.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:36.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509474 2024-08-19T14:41:36.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2024-08-19T14:41:36.193 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509474 2024-08-19T14:41:36.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:41:36.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509474 -lt 64424509474 2024-08-19T14:41:36.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:36.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070466 2024-08-19T14:41:36.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:36.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:41:36.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070466 2024-08-19T14:41:36.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:36.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070466 2024-08-19T14:41:36.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070466' 2024-08-19T14:41:36.461 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.3 seq 253403070466 2024-08-19T14:41:36.461 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:41:36.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070466 -lt 253403070466 2024-08-19T14:41:36.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:41:36.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:41:36.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:41:37.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:41:37.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:41:37.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:41:37.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:41:37.100 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:41:37.100 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:41:37.100 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:41:37.100 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:41:37.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:41:37.363 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:41:37.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:41:37.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:41:37.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:41:37.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:41:37.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:41:37.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T14:41:37.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T14:41:37.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-08-19T14:41:37.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-08-19T14:41:37.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-08-19T14:41:37.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-08-19T14:41:37.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-08-19T14:41:37.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-08-19T14:41:37.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:41:37.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:41:37.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:41:37.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 153261"' 2024-08-19T14:41:37.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 153261' 2024-08-19T14:41:37.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-08-19T14:41:37.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:41:37.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:41:37.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 153263"' 2024-08-19T14:41:37.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 153263' 2024-08-19T14:41:37.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:41:37.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-08-19T14:41:37.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 153261 153263' 2024-08-19T14:41:37.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-08-19T14:41:37.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:41:37.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 153261 2024-08-19T14:41:37.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/153264: /' 2024-08-19T14:41:37.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-08-19T14:41:37.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/153266: /' 2024-08-19T14:41:40.538 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:41:40.538 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:41:40.539 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T14:41:40.539 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:41:40.539 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:41:40.539 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:41:40.539 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:41:40.539 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T14:41:40.539 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:41:40.539 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T14:41:40.539 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:41:40.539 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:41:40.539 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:41:40.540 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:41:40.540 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:41:40.540 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:41:40.540 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:41:40.540 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:41:40.540 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:41:40.540 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T14:41:40.540 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:41:40.540 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:41:40.540 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-08-19T14:41:40.540 INFO:tasks.workunit.client.0.smithi102.stderr:153264: remove 1#2:eb822e21:::SOMETHING:head# 2024-08-19T14:41:40.540 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T14:41:40.540 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:41:40.540 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:41:40.541 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:41:40.541 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:41:40.541 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:41:40.541 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:41:40.541 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:41:40.541 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:41:40.542 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:41:40.542 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:41:40.542 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:41:40.542 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:41:40.542 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:41:40.542 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:41:40.542 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:41:40.542 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:41:40.542 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:41:40.542 INFO:tasks.workunit.client.0.smithi102.stderr:153264: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:41:40.542 INFO:tasks.workunit.client.0.smithi102.stderr:153264: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:41:40.542 INFO:tasks.workunit.client.0.smithi102.stderr:153264: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:41:40.543 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:41:40.543 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:41:40.543 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:41:40.543 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:41:40.543 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:41:40.543 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:41:40.543 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:41:40.543 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:41:40.544 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:41:40.544 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:41:40.544 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:41:40.544 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:41:40.544 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:41:40.544 INFO:tasks.workunit.client.0.smithi102.stderr:153264: start osd.1 2024-08-19T14:41:40.544 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:41:40.580 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:41:40.581 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:41:40.581 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-08-19T14:41:40.581 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:41:40.581 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-08-19T14:41:40.581 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:41:40.581 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:41:40.581 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-08-19T14:41:40.581 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:41:40.581 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-08-19T14:41:40.581 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:41:40.581 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:41:40.581 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:41:40.582 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:41:40.582 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:41:40.582 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:41:40.582 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-08-19T14:41:40.582 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:41:40.582 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:41:40.582 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-08-19T14:41:40.582 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:41:40.582 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-08-19T14:41:40.582 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-08-19T14:41:40.582 INFO:tasks.workunit.client.0.smithi102.stderr:153266: remove 2#2:eb822e21:::SOMETHING:head# 2024-08-19T14:41:40.582 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-08-19T14:41:40.582 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:41:40.583 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:41:40.583 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-19T14:41:40.583 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:41:40.583 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T14:41:40.583 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:41:40.583 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:41:40.584 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:41:40.584 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:41:40.584 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T14:41:40.584 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T14:41:40.584 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:41:40.584 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:41:40.584 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:41:40.584 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:41:40.584 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:41:40.584 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:41:40.584 INFO:tasks.workunit.client.0.smithi102.stderr:153266: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:41:40.584 INFO:tasks.workunit.client.0.smithi102.stderr:153266: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:41:40.584 INFO:tasks.workunit.client.0.smithi102.stderr:153266: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:41:40.585 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:41:40.585 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:41:40.585 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:41:40.585 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:41:40.585 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:41:40.585 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:41:40.585 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:41:40.585 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:41:40.585 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:41:40.585 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:41:40.585 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:41:40.585 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T14:41:40.585 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-19T14:41:40.586 INFO:tasks.workunit.client.0.smithi102.stderr:153266: start osd.2 2024-08-19T14:41:40.586 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:41:45.036 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T14:41:45.036 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:41:45.037 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:41:45.037 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:41:45.037 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:41:45.037 INFO:tasks.workunit.client.0.smithi102.stderr:153264: 2024-08-19T14:41:40.570+0000 7f60a643a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:41:45.037 INFO:tasks.workunit.client.0.smithi102.stderr:153264: 2024-08-19T14:41:40.573+0000 7f60a643a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:41:45.037 INFO:tasks.workunit.client.0.smithi102.stderr:153264: 2024-08-19T14:41:40.574+0000 7f60a643a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:41:45.037 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:41:45.037 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:41:45.037 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:41:45.037 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:41:45.037 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:41:45.037 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:41:45.037 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:41:45.038 INFO:tasks.workunit.client.0.smithi102.stderr:153264: 0 2024-08-19T14:41:45.038 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:41:45.038 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:41:45.038 INFO:tasks.workunit.client.0.smithi102.stderr:153264: 2024-08-19T14:41:41.093+0000 7f60a643a640 -1 Falling back to public interface 2024-08-19T14:41:45.038 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:41:45.038 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:41:45.038 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:41:45.038 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:41:45.038 INFO:tasks.workunit.client.0.smithi102.stderr:153264: 1 2024-08-19T14:41:45.038 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:41:45.038 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:41:45.038 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:41:45.038 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:41:45.039 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:41:45.039 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:41:45.039 INFO:tasks.workunit.client.0.smithi102.stderr:153264: 2 2024-08-19T14:41:45.039 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:41:45.039 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:41:45.039 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:41:45.039 INFO:tasks.workunit.client.0.smithi102.stderr:153264: 2024-08-19T14:41:43.820+0000 7f60a643a640 -1 osd.1 60 log_to_monitors true 2024-08-19T14:41:45.039 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:41:45.039 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:41:45.039 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:41:45.039 INFO:tasks.workunit.client.0.smithi102.stderr:153264: 3 2024-08-19T14:41:45.039 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:41:45.039 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:41:45.306 INFO:tasks.workunit.client.0.smithi102.stderr:153264: osd.1 up in weight 1 up_from 65 up_thru 56 down_at 61 last_153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-08-19T14:41:45.306 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-19T14:41:45.306 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:41:45.306 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:41:45.306 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:41:45.306 INFO:tasks.workunit.client.0.smithi102.stderr:153266: 2024-08-19T14:41:40.612+0000 7f0389db1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:41:45.306 INFO:tasks.workunit.client.0.smithi102.stderr:153266: 2024-08-19T14:41:40.614+0000 7f0389db1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:41:45.307 INFO:tasks.workunit.client.0.smithi102.stderr:153266: 2024-08-19T14:41:40.615+0000 7f0389db1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:41:45.307 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-19T14:41:45.307 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:41:45.307 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T14:41:45.307 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:41:45.307 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:41:45.307 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:41:45.307 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:41:45.307 INFO:tasks.workunit.client.0.smithi102.stderr:153266: 0 2024-08-19T14:41:45.307 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:41:45.307 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:41:45.307 INFO:tasks.workunit.client.0.smithi102.stderr:153266: 2024-08-19T14:41:41.130+0000 7f0389db1640 -1 Falling back to public interface 2024-08-19T14:41:45.307 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:41:45.307 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:41:45.308 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:41:45.308 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:41:45.308 INFO:tasks.workunit.client.0.smithi102.stderr:153266: 1 2024-08-19T14:41:45.308 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:41:45.308 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:41:45.308 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:41:45.308 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:41:45.308 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:41:45.308 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:41:45.308 INFO:tasks.workunit.client.0.smithi102.stderr:153266: 2 2024-08-19T14:41:45.308 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:41:45.308 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:41:45.308 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:41:45.308 INFO:tasks.workunit.client.0.smithi102.stderr:153266: 2024-08-19T14:41:44.006+0000 7f0389db1640 -1 osd.2 60 log_to_monitors true 2024-08-19T14:41:45.309 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:41:45.309 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:41:45.309 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:41:45.309 INFO:tasks.workunit.client.0.smithi102.stderr:153266: 3 2024-08-19T14:41:45.309 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:41:45.309 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:41:45.927 INFO:tasks.workunit.client.0.smithi102.stderr:153266: osd.2 up in weight 1 up_from 66 up_thru 49 down_at 61 last_clean_interval [53,60) [v2:127.0.0.1:6810/3818278930,v1:127.0.0.1:6811/3818278930] [v2:127.0.0.1:6812/3818278930,v1:127.0.0.1:6813/3818278930] exists,up 118c3f6f-c0bc-46e6-b0d9-b63cb0e35aeb 2024-08-19T14:41:45.927 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:41:45.927 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:41:45.927 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:41:45.927 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:41:45.927 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:41:45.927 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:41:45.927 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:41:45.927 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:41:45.927 INFO:tasks.workunit.client.0.smithi102.stderr:153264: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:41:45.928 INFO:tasks.workunit.client.0.smithi102.stderr:153264: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:41:45.928 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:41:45.928 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:41:45.928 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:41:45.928 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:41:45.928 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:41:45.928 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:41:45.928 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:41:45.928 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:41:45.928 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:41:45.928 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:41:45.928 INFO:tasks.workunit.client.0.smithi102.stderr:153264: 1 2024-08-19T14:41:45.928 INFO:tasks.workunit.client.0.smithi102.stderr:153264: 2 2024-08-19T14:41:45.929 INFO:tasks.workunit.client.0.smithi102.stderr:153264: 3' 2024-08-19T14:41:45.929 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:41:45.929 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:45.929 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:41:45.929 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836526 2024-08-19T14:41:45.929 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836526 2024-08-19T14:41:45.929 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836526' 2024-08-19T14:41:45.929 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:45.929 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:41:45.930 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874242 2024-08-19T14:41:45.930 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874242 2024-08-19T14:41:45.930 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836526 1-279172874242' 2024-08-19T14:41:45.930 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:46.114 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph clean_interval [15,60) [v2:127.0.0.1:6818/3977784967,v1:127.0.0.1:6819/3977784967] [v2:127.0.0.1:6820/3977784967,v1:127.0.0.1:6821/3977784967] exists,up 30952570-c55e-4456-b25b-267985f3e7cc 2024-08-19T14:41:46.114 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:41:46.115 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:41:46.115 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:41:46.115 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:41:46.115 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:41:46.115 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:41:46.115 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:41:46.115 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:41:46.115 INFO:tasks.workunit.client.0.smithi102.stderr:153266: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:41:46.115 INFO:tasks.workunit.client.0.smithi102.stderr:153266: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:41:46.116 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:41:46.116 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:41:46.116 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:41:46.116 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:41:46.116 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:41:46.116 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:41:46.116 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:41:46.116 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:41:46.116 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:41:46.116 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:41:46.116 INFO:tasks.workunit.client.0.smithi102.stderr:153266: 1 2024-08-19T14:41:46.116 INFO:tasks.workunit.client.0.smithi102.stderr:153266: 2 2024-08-19T14:41:46.116 INFO:tasks.workunit.client.0.smithi102.stderr:153266: 3' 2024-08-19T14:41:46.117 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:41:46.117 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:46.117 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:41:46.117 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836527 2024-08-19T14:41:46.117 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836527 2024-08-19T14:41:46.117 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527' 2024-08-19T14:41:46.117 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:46.117 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:41:46.117 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874243 2024-08-19T14:41:46.117 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874243 2024-08-19T14:41:46.117 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-279172874243' 2024-08-19T14:41:46.117 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:48.255 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:41:48.255 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-08-19T14:41:48.255 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-08-19T14:41:48.255 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836526 1-279172874242 2-283467841538' 2024-08-19T14:41:48.255 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:48.255 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:41:48.255 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070469 2024-08-19T14:41:48.255 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070469 2024-08-19T14:41:48.256 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836526 1-279172874242 2-283467841538 3-253403070469' 2024-08-19T14:41:48.256 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:48.256 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836526 2024-08-19T14:41:48.256 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:48.256 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:41:48.256 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836526 2024-08-19T14:41:48.256 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:48.256 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836526 2024-08-19T14:41:48.256 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-08-19T14:41:48.256 INFO:tasks.workunit.client.0.smithi102.stderr:153264: waiting osd.0 seq 21474836526 2024-08-19T14:41:48.256 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:41:48.256 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836524 -lt 21474836526 2024-08-19T14:41:48.256 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:41:48.257 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:41:48.257 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:41:48.257 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836527 -lt 21474836526 2024-08-19T14:41:48.257 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:48.257 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-279172874242 2024-08-19T14:41:48.257 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:48.257 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:41:48.257 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-279172874242 2024-08-19T14:41:48.257 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:48.257 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874242 2024-08-19T14:41:48.257 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 279172874242' 2024-08-19T14:41:48.257 INFO:tasks.workunit.client.0.smithi102.stderr:153264: waiting osd.1 seq 279172874242 2024-08-19T14:41:48.257 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:41:48.257 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874243 -lt 279172874242 2024-08-19T14:41:48.258 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:48.407 INFO:tasks.workunit.client.0.smithi102.stderr:1532tell osd.2 flush_pg_stats 2024-08-19T14:41:48.407 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841539 2024-08-19T14:41:48.407 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841539 2024-08-19T14:41:48.407 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-279172874243 2-283467841539' 2024-08-19T14:41:48.408 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:41:48.408 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:41:48.408 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070470 2024-08-19T14:41:48.408 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070470 2024-08-19T14:41:48.408 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-279172874243 2-283467841539 3-253403070470' 2024-08-19T14:41:48.408 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:48.408 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836527 2024-08-19T14:41:48.408 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:48.408 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:41:48.408 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836527 2024-08-19T14:41:48.408 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:48.408 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836527 2024-08-19T14:41:48.408 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-08-19T14:41:48.409 INFO:tasks.workunit.client.0.smithi102.stderr:153266: waiting osd.0 seq 21474836527 2024-08-19T14:41:48.409 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:41:48.409 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836524 -lt 21474836527 2024-08-19T14:41:48.409 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:41:48.409 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:41:48.409 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:41:48.409 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836527 -lt 21474836527 2024-08-19T14:41:48.409 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:48.409 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-279172874243 2024-08-19T14:41:48.409 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:48.409 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:41:48.409 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-279172874243 2024-08-19T14:41:48.409 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:48.409 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874243 2024-08-19T14:41:48.410 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 279172874243' 2024-08-19T14:41:48.410 INFO:tasks.workunit.client.0.smithi102.stderr:153266: waiting osd.1 seq 279172874243 2024-08-19T14:41:48.410 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:41:48.410 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874243 -lt 279172874243 2024-08-19T14:41:48.410 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:49.548 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-283467841538 2024-08-19T14:41:49.548 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:49.548 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:41:49.548 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-283467841538 2024-08-19T14:41:49.548 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:49.548 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-08-19T14:41:49.549 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 283467841538' 2024-08-19T14:41:49.549 INFO:tasks.workunit.client.0.smithi102.stderr:153264: waiting osd.2 seq 283467841538 2024-08-19T14:41:49.549 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:41:49.549 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841539 -lt 283467841538 2024-08-19T14:41:49.549 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:49.549 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070469 2024-08-19T14:41:49.549 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:49.549 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:41:49.549 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070469 2024-08-19T14:41:49.549 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:49.549 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070469 2024-08-19T14:41:49.549 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070469' 2024-08-19T14:41:49.549 INFO:tasks.workunit.client.0.smithi102.stderr:153264: waiting osd.3 seq 253403070469 2024-08-19T14:41:49.550 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:41:49.550 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070470 -lt 253403070469 2024-08-19T14:41:49.550 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:41:49.550 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:41:49.550 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:41:49.550 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:41:49.550 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:41:49.550 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:41:49.550 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:41:49.550 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:41:49.550 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:41:49.550 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:41:49.551 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:41:49.551 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-08-19T14:41:49.551 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:41:49.711 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.c66: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-283467841539 2024-08-19T14:41:49.711 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:49.711 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:41:49.712 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-283467841539 2024-08-19T14:41:49.712 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:49.712 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841539 2024-08-19T14:41:49.712 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 283467841539' 2024-08-19T14:41:49.712 INFO:tasks.workunit.client.0.smithi102.stderr:153266: waiting osd.2 seq 283467841539 2024-08-19T14:41:49.712 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:41:49.712 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841539 -lt 283467841539 2024-08-19T14:41:49.712 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:41:49.712 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070470 2024-08-19T14:41:49.712 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:41:49.712 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:41:49.712 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070470 2024-08-19T14:41:49.712 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:41:49.713 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070470 2024-08-19T14:41:49.713 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070470' 2024-08-19T14:41:49.713 INFO:tasks.workunit.client.0.smithi102.stderr:153266: waiting osd.3 seq 253403070470 2024-08-19T14:41:49.713 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:41:49.713 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070470 -lt 253403070470 2024-08-19T14:41:49.713 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:41:49.713 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:41:49.713 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:41:49.713 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:41:49.713 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:41:49.713 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:41:49.713 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:41:49.713 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:41:49.714 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:41:49.714 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:41:49.714 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:41:49.714 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-08-19T14:41:49.714 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:41:50.961 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:41:50.961 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:41:50.961 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-08-19T14:41:50.961 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-08-19T14:41:50.961 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:41:50.961 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-08-19T14:41:50.961 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:41:50.961 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:41:50.961 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:41:50.962 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:41:50.962 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:41:50.962 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:41:50.962 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:41:50.962 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:41:50.962 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:41:50.962 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:41:50.962 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-08-19T14:41:50.962 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:41:50.962 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:41:50.962 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:41:50.962 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-08-19T14:41:50.962 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-08-19T14:41:50.963 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T14:41:50.963 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T14:41:50.963 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T14:41:50.963 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T14:41:50.963 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T14:41:50.963 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T14:41:50.963 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T14:41:50.963 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T14:41:50.963 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T14:41:51.145 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/lient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:41:51.145 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:41:51.146 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-08-19T14:41:51.146 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-08-19T14:41:51.146 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:41:51.146 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-08-19T14:41:51.146 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:41:51.146 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:41:51.146 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:41:51.146 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:41:51.146 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:41:51.146 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:41:51.146 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:41:51.146 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:41:51.147 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:41:51.147 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:41:51.147 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-08-19T14:41:51.147 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:41:51.147 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:41:51.147 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:41:51.147 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-08-19T14:41:51.147 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-08-19T14:41:51.147 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T14:41:51.147 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T14:41:51.147 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T14:41:51.147 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T14:41:51.147 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T14:41:51.147 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T14:41:51.148 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T14:41:51.148 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T14:41:51.148 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T14:41:51.770 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-19T14:41:51.771 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:41:51.771 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-19T14:41:51.771 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:41:51.771 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:41:51.771 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:41:51.771 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:41:51.771 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:41:51.771 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:41:51.771 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:41:51.771 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:41:51.771 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:41:51.771 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:41:51.772 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:41:51.772 INFO:tasks.workunit.client.0.smithi102.stderr:153264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:41:51.772 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:41:51.772 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:41:51.772 INFO:tasks.workunit.client.0.smithi102.stderr:153264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:41:51.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:41:51.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:41:51.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 153263 2024-08-19T14:41:51.953 INFO:tasks.workunit.client.0.smithi102.stderr:clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-19T14:41:51.953 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:41:51.953 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-19T14:41:51.954 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:41:51.954 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:41:51.954 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:41:51.954 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:41:51.954 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:41:51.954 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:41:51.954 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:41:51.954 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:41:51.954 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:41:51.954 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:41:51.954 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:41:51.954 INFO:tasks.workunit.client.0.smithi102.stderr:153266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:41:51.955 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:41:51.955 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:41:51.955 INFO:tasks.workunit.client.0.smithi102.stderr:153266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:41:51.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:41:51.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-08-19T14:41:51.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-08-19T14:41:51.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-08-19T14:41:51.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-08-19T14:41:51.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-08-19T14:41:51.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-08-19T14:41:51.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T14:41:51.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T14:41:51.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:41:51.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:41:52.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-08-19T14:41:52.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-08-19T14:41:52.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-08-19T14:41:52.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-08-19T14:41:52.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:41:52.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:41:52.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:41:52.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:41:52.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T14:41:24.317930+0000 2024-08-19T14:41:52.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-08-19T14:41:52.763 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-08-19T14:41:52.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-08-19T14:41:24.317930+0000 2024-08-19T14:41:52.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T14:41:52.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:41:24.317930+0000 2024-08-19T14:41:52.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:41:52.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:41:52.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:41:52.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:41:52.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:41:52.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:41:52.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:41:52.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:41:53.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:41:24.317930+0000 '>' 2024-08-19T14:41:24.317930+0000 2024-08-19T14:41:53.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:41:54.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:41:54.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:41:54.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:41:54.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:41:54.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:41:54.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:41:54.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:41:54.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:41:24.317930+0000 '>' 2024-08-19T14:41:24.317930+0000 2024-08-19T14:41:54.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:41:55.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:41:55.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:41:55.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:41:55.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:41:55.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:41:55.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:41:55.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:41:55.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:41:24.317930+0000 '>' 2024-08-19T14:41:24.317930+0000 2024-08-19T14:41:55.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:41:56.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:41:56.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:41:56.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:41:56.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:41:56.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:41:56.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:41:56.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:41:56.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:41:24.317930+0000 '>' 2024-08-19T14:41:24.317930+0000 2024-08-19T14:41:56.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:41:57.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:41:57.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:41:57.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:41:57.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:41:57.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:41:57.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:41:57.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:41:58.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:41:24.317930+0000 '>' 2024-08-19T14:41:24.317930+0000 2024-08-19T14:41:58.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:41:59.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:41:59.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:41:59.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:41:59.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:41:59.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:41:59.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:41:59.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:41:59.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:41:53.739238+0000 '>' 2024-08-19T14:41:24.317930+0000 2024-08-19T14:41:59.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:41:59.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-08-19T14:41:59.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:41:59.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:41:59.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:41:59.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 156382"' 2024-08-19T14:41:59.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 156382' 2024-08-19T14:41:59.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-08-19T14:41:59.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:41:59.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:41:59.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 156384"' 2024-08-19T14:41:59.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:41:59.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 156384' 2024-08-19T14:41:59.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-08-19T14:41:59.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 156382 156384' 2024-08-19T14:41:59.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-08-19T14:41:59.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:41:59.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/156385: /' 2024-08-19T14:41:59.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 156382 2024-08-19T14:41:59.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-08-19T14:41:59.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/156387: /' 2024-08-19T14:42:01.973 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:42:01.973 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:42:01.973 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-08-19T14:42:01.973 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:42:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-08-19T14:42:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:42:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:42:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-08-19T14:42:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:42:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-08-19T14:42:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:42:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:42:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:42:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:42:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:42:01.975 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:42:01.975 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-08-19T14:42:01.975 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:42:01.975 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:42:01.975 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-08-19T14:42:01.975 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:42:01.975 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-08-19T14:42:01.975 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-08-19T14:42:01.976 INFO:tasks.workunit.client.0.smithi102.stderr:156387: _ 2024-08-19T14:42:01.976 INFO:tasks.workunit.client.0.smithi102.stderr:156387: hinfo_key 2024-08-19T14:42:01.976 INFO:tasks.workunit.client.0.smithi102.stderr:156387: snapset 2024-08-19T14:42:01.976 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-08-19T14:42:01.976 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:42:01.976 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:42:01.976 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-19T14:42:01.976 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:42:01.976 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T14:42:01.976 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:42:01.976 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:42:01.977 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:42:01.977 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:42:01.977 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T14:42:01.977 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T14:42:01.977 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:42:01.977 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:42:01.977 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:42:01.977 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:42:01.977 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:42:01.977 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:42:01.978 INFO:tasks.workunit.client.0.smithi102.stderr:156387: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:42:01.978 INFO:tasks.workunit.client.0.smithi102.stderr:156387: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:42:01.978 INFO:tasks.workunit.client.0.smithi102.stderr:156387: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:42:01.978 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:42:01.978 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:42:01.978 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:42:01.978 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:42:01.978 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:42:01.978 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:42:01.978 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:42:01.979 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:42:01.979 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:42:01.979 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:42:01.979 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:42:01.979 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T14:42:01.979 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-19T14:42:01.979 INFO:tasks.workunit.client.0.smithi102.stderr:156387: start osd.2 2024-08-19T14:42:02.009 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:42:02.010 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:42:02.010 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T14:42:02.010 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:42:02.010 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:42:02.010 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:42:02.010 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:42:02.010 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T14:42:02.010 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:42:02.010 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T14:42:02.010 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:42:02.011 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:42:02.011 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:42:02.011 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:42:02.011 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:42:02.011 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:42:02.011 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:42:02.011 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:42:02.011 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:42:02.011 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T14:42:02.011 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:42:02.011 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:42:02.011 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-08-19T14:42:02.011 INFO:tasks.workunit.client.0.smithi102.stderr:156385: _ 2024-08-19T14:42:02.012 INFO:tasks.workunit.client.0.smithi102.stderr:156385: hinfo_key 2024-08-19T14:42:02.012 INFO:tasks.workunit.client.0.smithi102.stderr:156385: snapset 2024-08-19T14:42:02.012 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T14:42:02.012 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:42:02.012 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:42:02.012 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:42:02.012 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:42:02.012 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:42:02.012 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:42:02.012 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:42:02.013 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:42:02.013 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:42:02.013 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:42:02.013 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:42:02.013 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:42:02.013 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:42:02.013 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:42:02.013 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:42:02.014 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:42:02.014 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:42:02.014 INFO:tasks.workunit.client.0.smithi102.stderr:156385: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:42:02.014 INFO:tasks.workunit.client.0.smithi102.stderr:156385: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:42:02.014 INFO:tasks.workunit.client.0.smithi102.stderr:156385: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:42:02.014 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:42:02.014 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:42:02.014 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:42:02.014 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:42:02.014 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:42:02.014 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:42:02.014 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:42:02.015 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:42:02.015 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:42:02.015 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:42:02.015 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:42:02.015 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:42:02.015 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:42:02.015 INFO:tasks.workunit.client.0.smithi102.stderr:156385: start osd.1 2024-08-19T14:42:06.503 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:42:06.503 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T14:42:06.503 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:42:06.503 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:42:06.503 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:42:06.503 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:42:06.503 INFO:tasks.workunit.client.0.smithi102.stderr:156385: 2024-08-19T14:42:02.039+0000 7fe26980e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:06.503 INFO:tasks.workunit.client.0.smithi102.stderr:156385: 2024-08-19T14:42:02.042+0000 7fe26980e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:06.503 INFO:tasks.workunit.client.0.smithi102.stderr:156385: 2024-08-19T14:42:02.043+0000 7fe26980e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:06.503 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:42:06.503 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:42:06.503 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:42:06.504 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:42:06.504 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:42:06.504 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:06.504 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:42:06.504 INFO:tasks.workunit.client.0.smithi102.stderr:156385: 0 2024-08-19T14:42:06.504 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:06.504 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:42:06.504 INFO:tasks.workunit.client.0.smithi102.stderr:156385: 2024-08-19T14:42:02.571+0000 7fe26980e640 -1 Falling back to public interface 2024-08-19T14:42:06.504 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:06.504 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:06.504 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:06.504 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:42:06.504 INFO:tasks.workunit.client.0.smithi102.stderr:156385: 1 2024-08-19T14:42:06.505 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:06.505 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:42:06.505 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:06.505 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:06.505 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:06.505 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:42:06.505 INFO:tasks.workunit.client.0.smithi102.stderr:156385: 2 2024-08-19T14:42:06.505 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:06.505 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:42:06.505 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:06.505 INFO:tasks.workunit.client.0.smithi102.stderr:156385: 2024-08-19T14:42:05.248+0000 7fe26980e640 -1 osd.1 69 log_to_monitors true 2024-08-19T14:42:06.505 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:06.505 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:06.505 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:42:06.506 INFO:tasks.workunit.client.0.smithi102.stderr:156385: 3 2024-08-19T14:42:06.506 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:06.506 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:42:06.506 INFO:tasks.workunit.client.0.smithi102.stderr:156385: osd.1 up in weight 1 up_from 75 up_thru 66 down_aovery_ops 2024-08-19T14:42:06.506 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-08-19T14:42:06.506 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-19T14:42:06.506 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:42:06.506 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:42:06.506 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:42:06.506 INFO:tasks.workunit.client.0.smithi102.stderr:156387: 2024-08-19T14:42:02.004+0000 7fbc3520f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:06.506 INFO:tasks.workunit.client.0.smithi102.stderr:156387: 2024-08-19T14:42:02.007+0000 7fbc3520f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:06.506 INFO:tasks.workunit.client.0.smithi102.stderr:156387: 2024-08-19T14:42:02.008+0000 7fbc3520f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:06.506 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-19T14:42:06.507 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:42:06.507 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T14:42:06.507 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:42:06.507 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:42:06.507 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:06.507 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:42:06.507 INFO:tasks.workunit.client.0.smithi102.stderr:156387: 0 2024-08-19T14:42:06.507 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:06.507 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:42:06.507 INFO:tasks.workunit.client.0.smithi102.stderr:156387: 2024-08-19T14:42:02.524+0000 7fbc3520f640 -1 Falling back to public interface 2024-08-19T14:42:06.507 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:06.507 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:06.507 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:06.508 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:42:06.508 INFO:tasks.workunit.client.0.smithi102.stderr:156387: 1 2024-08-19T14:42:06.508 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:06.508 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:42:06.508 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:06.508 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:06.508 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:06.508 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:42:06.509 INFO:tasks.workunit.client.0.smithi102.stderr:156387: 2 2024-08-19T14:42:06.509 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:06.509 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:42:06.509 INFO:tasks.workunit.client.0.smithi102.stderr:156387: 2024-08-19T14:42:05.081+0000 7fbc3520f640 -1 osd.2 69 log_to_monitors true 2024-08-19T14:42:06.509 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:06.509 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:06.509 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:06.509 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:42:06.509 INFO:tasks.workunit.client.0.smithi102.stderr:156387: 3 2024-08-19T14:42:06.509 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:06.509 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:42:07.325 INFO:tasks.workunit.client.0.smithi102.stderr:156387: osd.2 up in weight 1 up_from 74 up_thru 74 down_at 70 last_clean_interval [65,69) [v2:127.0.0.1:6818/3000180120,v1:127.0.0.1:6819/3000180120] [v2:127.0.0.1:6820/3000180120,v1:127.0.0.1:6821/3000180120] exists,up 118c3f6f-c0bc-46e6-b0d9-b63cb0e35aeb 2024-08-19T14:42:07.326 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:42:07.326 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:42:07.326 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:42:07.326 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:42:07.326 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:42:07.326 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:42:07.326 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:42:07.326 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:42:07.326 INFO:tasks.workunit.client.0.smithi102.stderr:156385: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:42:07.326 INFO:tasks.workunit.client.0.smithi102.stderr:156385: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:42:07.327 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:42:07.327 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:42:07.327 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:42:07.327 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:42:07.327 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:42:07.327 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:42:07.327 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:42:07.327 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:42:07.327 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:42:07.327 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:42:07.327 INFO:tasks.workunit.client.0.smithi102.stderr:156385: 1 2024-08-19T14:42:07.327 INFO:tasks.workunit.client.0.smithi102.stderr:156385: 2 2024-08-19T14:42:07.327 INFO:tasks.workunit.client.0.smithi102.stderr:156385: 3' 2024-08-19T14:42:07.327 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:42:07.328 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:07.328 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:42:07.328 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836533 2024-08-19T14:42:07.328 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836533 2024-08-19T14:42:07.328 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533' 2024-08-19T14:42:07.328 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:07.328 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:42:07.328 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547202 2024-08-19T14:42:07.328 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547202 2024-08-19T14:42:07.328 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533 1-322122547202' 2024-08-19T14:42:07.328 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:07.328 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stat 70 last_clean_interval [66,69) [v2:127.0.0.1:6810/1757531792,v1:127.0.0.1:6811/1757531792] [v2:127.0.0.1:6812/1757531792,v1:127.0.0.1:6813/1757531792] exists,up 30952570-c55e-4456-b25b-267985f3e7cc 2024-08-19T14:42:07.328 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:42:07.329 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:42:07.329 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:42:07.329 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:42:07.329 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:42:07.329 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:42:07.329 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:42:07.329 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:42:07.329 INFO:tasks.workunit.client.0.smithi102.stderr:156387: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:42:07.329 INFO:tasks.workunit.client.0.smithi102.stderr:156387: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:42:07.329 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:42:07.329 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:42:07.329 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:42:07.329 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:42:07.330 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:42:07.330 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:42:07.330 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:42:07.330 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:42:07.330 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:42:07.330 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:42:07.330 INFO:tasks.workunit.client.0.smithi102.stderr:156387: 1 2024-08-19T14:42:07.330 INFO:tasks.workunit.client.0.smithi102.stderr:156387: 2 2024-08-19T14:42:07.330 INFO:tasks.workunit.client.0.smithi102.stderr:156387: 3' 2024-08-19T14:42:07.330 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:42:07.330 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:07.330 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:42:07.331 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836532 2024-08-19T14:42:07.331 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836532 2024-08-19T14:42:07.331 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532' 2024-08-19T14:42:07.331 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:07.331 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:42:07.331 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547203 2024-08-19T14:42:07.331 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547203 2024-08-19T14:42:07.331 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-322122547203' 2024-08-19T14:42:07.331 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:09.465 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:42:09.466 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-08-19T14:42:09.466 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-08-19T14:42:09.466 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533 1-322122547202 2-317827579906' 2024-08-19T14:42:09.466 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:09.466 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:42:09.466 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070476 2024-08-19T14:42:09.466 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070476 2024-08-19T14:42:09.466 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533 1-322122547202 2-317827579906 3-253403070476' 2024-08-19T14:42:09.466 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:09.466 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836533 2024-08-19T14:42:09.466 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:09.467 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:42:09.467 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836533 2024-08-19T14:42:09.467 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:09.467 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836533 2024-08-19T14:42:09.467 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2024-08-19T14:42:09.467 INFO:tasks.workunit.client.0.smithi102.stderr:156385: waiting osd.0 seq 21474836533 2024-08-19T14:42:09.467 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:42:09.467 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836533 -lt 21474836533 2024-08-19T14:42:09.467 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:09.467 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-322122547202 2024-08-19T14:42:09.467 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:09.467 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:42:09.467 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-322122547202 2024-08-19T14:42:09.468 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:09.468 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547202 2024-08-19T14:42:09.468 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 322122547202' 2024-08-19T14:42:09.468 INFO:tasks.workunit.client.0.smithi102.stderr:156385: waiting osd.1 seq 322122547202 2024-08-19T14:42:09.468 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:42:09.468 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 322122547202 2024-08-19T14:42:09.468 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:42:09.468 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:42:09.468 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:42:09.468 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547203 -lt 322122547202 2024-08-19T14:42:09.468 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:09.469 INFO:tasks.workunit.client.0.smithi102.stderr:156ts: ceph tell osd.2 flush_pg_stats 2024-08-19T14:42:09.469 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579907 2024-08-19T14:42:09.469 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579907 2024-08-19T14:42:09.469 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-322122547203 2-317827579907' 2024-08-19T14:42:09.469 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:09.469 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:42:09.469 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070475 2024-08-19T14:42:09.469 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070475 2024-08-19T14:42:09.469 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-322122547203 2-317827579907 3-253403070475' 2024-08-19T14:42:09.469 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:09.469 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836532 2024-08-19T14:42:09.469 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:09.470 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:42:09.470 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836532 2024-08-19T14:42:09.470 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:09.470 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836532 2024-08-19T14:42:09.470 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2024-08-19T14:42:09.470 INFO:tasks.workunit.client.0.smithi102.stderr:156387: waiting osd.0 seq 21474836532 2024-08-19T14:42:09.470 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:42:09.470 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836533 -lt 21474836532 2024-08-19T14:42:09.470 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:09.470 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-322122547203 2024-08-19T14:42:09.470 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:09.470 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:42:09.471 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-322122547203 2024-08-19T14:42:09.471 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:09.471 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547203 2024-08-19T14:42:09.471 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 322122547203' 2024-08-19T14:42:09.471 INFO:tasks.workunit.client.0.smithi102.stderr:156387: waiting osd.1 seq 322122547203 2024-08-19T14:42:09.471 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:42:09.471 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 322122547203 2024-08-19T14:42:09.471 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:42:09.471 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:42:09.471 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:42:09.471 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547203 -lt 322122547203 2024-08-19T14:42:09.471 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:10.640 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579906 2024-08-19T14:42:10.640 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:10.640 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:42:10.640 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579906 2024-08-19T14:42:10.640 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:10.640 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-08-19T14:42:10.640 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579906' 2024-08-19T14:42:10.640 INFO:tasks.workunit.client.0.smithi102.stderr:156385: waiting osd.2 seq 317827579906 2024-08-19T14:42:10.640 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:42:10.641 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579907 -lt 317827579906 2024-08-19T14:42:10.641 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:10.641 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070476 2024-08-19T14:42:10.641 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:10.641 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:42:10.641 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070476 2024-08-19T14:42:10.641 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:10.641 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070476 2024-08-19T14:42:10.641 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070476' 2024-08-19T14:42:10.641 INFO:tasks.workunit.client.0.smithi102.stderr:156385: waiting osd.3 seq 253403070476 2024-08-19T14:42:10.641 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:42:10.641 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070477 -lt 253403070476 2024-08-19T14:42:10.641 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:42:10.641 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:42:10.642 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:42:10.642 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:42:10.642 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:42:10.642 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:42:10.642 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:42:10.642 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:42:10.642 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:42:10.642 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:42:10.642 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:42:10.642 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-08-19T14:42:10.642 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:42:10.713 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579907 2024-08-19T14:42:10.713 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:10.713 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:42:10.714 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579907 2024-08-19T14:42:10.714 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:10.714 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579907 2024-08-19T14:42:10.714 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579907' 2024-08-19T14:42:10.714 INFO:tasks.workunit.client.0.smithi102.stderr:156387: waiting osd.2 seq 317827579907 2024-08-19T14:42:10.714 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:42:10.714 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579907 -lt 317827579907 2024-08-19T14:42:10.714 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:10.714 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070475 2024-08-19T14:42:10.714 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:10.714 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:42:10.714 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070475 2024-08-19T14:42:10.714 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070475 2024-08-19T14:42:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070475' 2024-08-19T14:42:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:156387: waiting osd.3 seq 253403070475 2024-08-19T14:42:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:42:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070477 -lt 253403070475 2024-08-19T14:42:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:42:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:42:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:42:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:42:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:42:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:42:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:42:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:42:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:42:10.716 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:42:10.716 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:42:10.716 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-08-19T14:42:10.716 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:42:11.696 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:42:11.696 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:42:11.697 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-08-19T14:42:11.697 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-08-19T14:42:11.697 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:42:11.697 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-08-19T14:42:11.697 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:42:11.697 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:42:11.697 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:42:11.697 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:42:11.697 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:42:11.697 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:42:11.697 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:42:11.697 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:42:11.697 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:42:11.697 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:42:11.698 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:42:11.698 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:42:11.698 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:42:11.698 INFO:tasks.workunit.client.0.smithi102.stderr:156385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:42:11.698 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:42:11.698 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:42:11.698 INFO:tasks.workunit.client.0.smithi102.stderr:156385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:42:11.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:42:11.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:42:11.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 156384 2024-08-19T14:42:11.820 INFO:tasks.workunit.client.0.smithi102.stderr:client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:42:11.820 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:42:11.820 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-08-19T14:42:11.820 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-08-19T14:42:11.820 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:42:11.820 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-08-19T14:42:11.820 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:42:11.820 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:42:11.820 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:42:11.820 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:42:11.820 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:42:11.821 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:42:11.821 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:42:11.821 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:42:11.821 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:42:11.821 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:42:11.821 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:42:11.821 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:42:11.821 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:42:11.821 INFO:tasks.workunit.client.0.smithi102.stderr:156387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:42:11.821 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:42:11.821 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:42:11.822 INFO:tasks.workunit.client.0.smithi102.stderr:156387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:42:11.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:42:11.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-08-19T14:42:11.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-08-19T14:42:11.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-08-19T14:42:11.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-08-19T14:42:11.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-08-19T14:42:11.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T14:42:11.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T14:42:11.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-08-19T14:42:11.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-08-19T14:42:11.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-08-19T14:42:11.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-08-19T14:42:11.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-08-19T14:42:11.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-08-19T14:42:11.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:42:11.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:42:11.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:42:11.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 159159"' 2024-08-19T14:42:11.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 159159' 2024-08-19T14:42:11.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:42:11.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:42:11.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:42:11.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 159161"' 2024-08-19T14:42:11.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:42:11.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 159161' 2024-08-19T14:42:11.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-08-19T14:42:11.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 159159 159161' 2024-08-19T14:42:11.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-08-19T14:42:11.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:42:11.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 159159 2024-08-19T14:42:11.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/159162: /' 2024-08-19T14:42:11.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:42:11.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/159164: /' 2024-08-19T14:42:14.460 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:42:14.460 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:42:14.460 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T14:42:14.460 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:42:14.461 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:42:14.461 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:42:14.461 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:42:14.461 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T14:42:14.461 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:42:14.461 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T14:42:14.461 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:42:14.461 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:42:14.461 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:42:14.461 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:42:14.461 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:42:14.461 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:42:14.461 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:42:14.462 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:42:14.462 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:42:14.462 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T14:42:14.462 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:42:14.462 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:42:14.462 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-08-19T14:42:14.462 INFO:tasks.workunit.client.0.smithi102.stderr:159164: remove 1#2:eb822e21:::SOMETHING:head# 2024-08-19T14:42:14.462 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T14:42:14.462 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:42:14.462 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:42:14.462 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:42:14.462 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:42:14.462 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:42:14.462 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:42:14.463 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:42:14.464 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:42:14.464 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:42:14.464 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:42:14.464 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:42:14.464 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:42:14.464 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:42:14.464 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:42:14.464 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:42:14.464 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:42:14.464 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:42:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:159164: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:42:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:159164: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:42:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:159164: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:42:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:42:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:42:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:42:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:42:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:42:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:42:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:42:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:42:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:42:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:42:14.466 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:42:14.466 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:42:14.466 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:42:14.466 INFO:tasks.workunit.client.0.smithi102.stderr:159164: start osd.1 2024-08-19T14:42:14.466 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:42:14.923 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:42:14.923 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:42:14.923 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-08-19T14:42:14.923 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:42:14.923 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:42:14.923 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:42:14.924 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:42:14.924 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-08-19T14:42:14.924 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:42:14.924 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-08-19T14:42:14.924 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:42:14.924 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:42:14.924 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:42:14.924 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:42:14.924 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:42:14.924 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:42:14.924 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:42:14.924 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:42:14.925 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:42:14.925 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-08-19T14:42:14.925 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:42:14.926 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:42:14.926 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-08-19T14:42:14.926 INFO:tasks.workunit.client.0.smithi102.stderr:159162: remove 0#2:eb822e21:::SOMETHING:head# 2024-08-19T14:42:14.926 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-08-19T14:42:14.926 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:42:14.926 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:42:14.926 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-19T14:42:14.926 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:42:14.926 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:42:14.926 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:42:14.926 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:42:14.927 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:42:14.927 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:42:14.927 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T14:42:14.927 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T14:42:14.927 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:42:14.927 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:42:14.927 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:42:14.928 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:42:14.928 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:42:14.928 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:42:14.928 INFO:tasks.workunit.client.0.smithi102.stderr:159162: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:42:14.928 INFO:tasks.workunit.client.0.smithi102.stderr:159162: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:42:14.928 INFO:tasks.workunit.client.0.smithi102.stderr:159162: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:42:14.928 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:42:14.928 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:42:14.928 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:42:14.929 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:42:14.929 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:42:14.929 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:42:14.929 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:42:14.929 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:42:14.929 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:42:14.929 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:42:14.929 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:42:14.929 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T14:42:14.929 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-19T14:42:14.929 INFO:tasks.workunit.client.0.smithi102.stderr:159162: start osd.3 2024-08-19T14:42:14.930 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:42:18.972 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T14:42:18.973 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:42:18.973 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:42:18.973 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:42:18.973 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:42:18.973 INFO:tasks.workunit.client.0.smithi102.stderr:159164: 2024-08-19T14:42:14.492+0000 7f8279333640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:18.973 INFO:tasks.workunit.client.0.smithi102.stderr:159164: 2024-08-19T14:42:14.495+0000 7f8279333640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:18.973 INFO:tasks.workunit.client.0.smithi102.stderr:159164: 2024-08-19T14:42:14.496+0000 7f8279333640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:18.973 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:42:18.973 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:42:18.973 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:42:18.974 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:42:18.974 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:42:18.974 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:18.974 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:42:18.974 INFO:tasks.workunit.client.0.smithi102.stderr:159164: 0 2024-08-19T14:42:18.974 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:18.974 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:42:18.974 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:18.974 INFO:tasks.workunit.client.0.smithi102.stderr:159164: 2024-08-19T14:42:15.015+0000 7f8279333640 -1 Falling back to public interface 2024-08-19T14:42:18.974 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:18.974 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:18.974 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:42:18.975 INFO:tasks.workunit.client.0.smithi102.stderr:159164: 1 2024-08-19T14:42:18.975 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:18.975 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:42:18.975 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:18.975 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:18.975 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:18.975 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:42:18.975 INFO:tasks.workunit.client.0.smithi102.stderr:159164: 2 2024-08-19T14:42:18.975 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:18.975 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:42:18.975 INFO:tasks.workunit.client.0.smithi102.stderr:159164: 2024-08-19T14:42:17.571+0000 7f8279333640 -1 osd.1 77 log_to_monitors true 2024-08-19T14:42:18.975 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:18.976 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:18.976 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:18.976 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:42:18.976 INFO:tasks.workunit.client.0.smithi102.stderr:159164: 3 2024-08-19T14:42:18.976 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:18.976 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:42:19.378 INFO:tasks.workunit.client.0.smithi102.stderr:159164: osd.1 up in weight 1 up_from 82 up_thru 82 down_at 78 last_159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-08-19T14:42:19.378 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-19T14:42:19.378 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:42:19.378 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:42:19.378 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:42:19.379 INFO:tasks.workunit.client.0.smithi102.stderr:159162: 2024-08-19T14:42:14.951+0000 7f0099d2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:19.379 INFO:tasks.workunit.client.0.smithi102.stderr:159162: 2024-08-19T14:42:14.954+0000 7f0099d2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:19.379 INFO:tasks.workunit.client.0.smithi102.stderr:159162: 2024-08-19T14:42:14.955+0000 7f0099d2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:19.379 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-19T14:42:19.379 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:42:19.379 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T14:42:19.379 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:42:19.379 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:42:19.379 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:19.379 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:42:19.380 INFO:tasks.workunit.client.0.smithi102.stderr:159162: 0 2024-08-19T14:42:19.380 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:19.380 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:42:19.380 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:19.380 INFO:tasks.workunit.client.0.smithi102.stderr:159162: 2024-08-19T14:42:15.488+0000 7f0099d2c640 -1 Falling back to public interface 2024-08-19T14:42:19.380 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:19.380 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:19.380 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:42:19.380 INFO:tasks.workunit.client.0.smithi102.stderr:159162: 1 2024-08-19T14:42:19.380 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:19.380 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:42:19.381 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:19.381 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:19.381 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:19.381 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:42:19.381 INFO:tasks.workunit.client.0.smithi102.stderr:159162: 2 2024-08-19T14:42:19.381 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:19.381 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:42:19.381 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:19.381 INFO:tasks.workunit.client.0.smithi102.stderr:159162: 2024-08-19T14:42:18.150+0000 7f0099d2c640 -1 osd.3 77 log_to_monitors true 2024-08-19T14:42:19.381 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:19.381 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:19.381 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:42:19.382 INFO:tasks.workunit.client.0.smithi102.stderr:159162: 3 2024-08-19T14:42:19.382 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:19.382 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:42:19.880 INFO:tasks.workunit.client.0.smithi102.stderr:159162: osd.3 up in weight 1 up_from 84 up_thru 84 down_at 78 last_clean_interval [75,77) [v2:127.0.0.1:6818/3336495781,v1:127.0.0.1:6819/3336495781] [v2:127.0.0.1:6820/3336495781,v1:127.0.0.1:6821/3336495781] exists,up 118c3f6f-c0bc-46e6-b0d9-b63cb0e35aeb 2024-08-19T14:42:19.880 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:42:19.881 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:42:19.881 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:42:19.881 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:42:19.881 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:42:19.881 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:42:19.881 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:42:19.881 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:42:19.881 INFO:tasks.workunit.client.0.smithi102.stderr:159164: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:42:19.881 INFO:tasks.workunit.client.0.smithi102.stderr:159164: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:42:19.881 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:42:19.881 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:42:19.881 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:42:19.881 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:42:19.882 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:42:19.882 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:42:19.882 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:42:19.882 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:42:19.882 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:42:19.882 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:42:19.882 INFO:tasks.workunit.client.0.smithi102.stderr:159164: 1 2024-08-19T14:42:19.882 INFO:tasks.workunit.client.0.smithi102.stderr:159164: 2 2024-08-19T14:42:19.882 INFO:tasks.workunit.client.0.smithi102.stderr:159164: 3' 2024-08-19T14:42:19.882 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:42:19.882 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:19.882 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:42:19.883 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836536 2024-08-19T14:42:19.883 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836536 2024-08-19T14:42:19.883 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536' 2024-08-19T14:42:19.883 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:19.883 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:42:19.883 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-08-19T14:42:19.883 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-08-19T14:42:19.883 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-352187318274' 2024-08-19T14:42:19.883 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:20.166 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph clean_interval [59,77) [v2:127.0.0.1:6826/3675375400,v1:127.0.0.1:6827/3675375400] [v2:127.0.0.1:6828/3675375400,v1:127.0.0.1:6829/3675375400] exists,up 11030134-f10d-4d38-9e2b-e4fab1d556d5 2024-08-19T14:42:20.166 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:42:20.167 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:42:20.167 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:42:20.167 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:42:20.167 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:42:20.167 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:42:20.167 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:42:20.167 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:42:20.167 INFO:tasks.workunit.client.0.smithi102.stderr:159162: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:42:20.167 INFO:tasks.workunit.client.0.smithi102.stderr:159162: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:42:20.167 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:42:20.167 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:42:20.167 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:42:20.167 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:42:20.168 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:42:20.168 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:42:20.168 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:42:20.168 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:42:20.168 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:42:20.168 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:42:20.168 INFO:tasks.workunit.client.0.smithi102.stderr:159162: 1 2024-08-19T14:42:20.168 INFO:tasks.workunit.client.0.smithi102.stderr:159162: 2 2024-08-19T14:42:20.168 INFO:tasks.workunit.client.0.smithi102.stderr:159162: 3' 2024-08-19T14:42:20.168 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:42:20.168 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:20.168 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:42:20.168 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836537 2024-08-19T14:42:20.169 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836537 2024-08-19T14:42:20.169 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537' 2024-08-19T14:42:20.169 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:20.169 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:42:20.169 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318275 2024-08-19T14:42:20.169 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318275 2024-08-19T14:42:20.169 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-352187318275' 2024-08-19T14:42:20.169 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:22.191 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:42:22.191 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579910 2024-08-19T14:42:22.191 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579910 2024-08-19T14:42:22.191 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-352187318274 2-317827579910' 2024-08-19T14:42:22.191 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:22.191 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:42:22.191 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252866 2024-08-19T14:42:22.191 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252866 2024-08-19T14:42:22.191 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-352187318274 2-317827579910 3-360777252866' 2024-08-19T14:42:22.191 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:22.191 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836536 2024-08-19T14:42:22.192 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:22.192 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:42:22.192 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836536 2024-08-19T14:42:22.192 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:22.192 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836536 2024-08-19T14:42:22.192 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2024-08-19T14:42:22.192 INFO:tasks.workunit.client.0.smithi102.stderr:159164: waiting osd.0 seq 21474836536 2024-08-19T14:42:22.192 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:42:22.192 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836535 -lt 21474836536 2024-08-19T14:42:22.192 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:42:22.192 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:42:22.192 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:42:22.193 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836536 2024-08-19T14:42:22.193 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:22.193 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-352187318274 2024-08-19T14:42:22.193 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:22.193 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:42:22.193 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-352187318274 2024-08-19T14:42:22.193 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:22.193 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-08-19T14:42:22.193 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 352187318274' 2024-08-19T14:42:22.193 INFO:tasks.workunit.client.0.smithi102.stderr:159164: waiting osd.1 seq 352187318274 2024-08-19T14:42:22.193 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:42:22.193 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318275 -lt 352187318274 2024-08-19T14:42:22.193 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:22.341 INFO:tasks.workunit.client.0.smithi102.stderr:1591tell osd.2 flush_pg_stats 2024-08-19T14:42:22.341 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579911 2024-08-19T14:42:22.341 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579911 2024-08-19T14:42:22.341 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-352187318275 2-317827579911' 2024-08-19T14:42:22.341 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:22.342 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:42:22.342 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252867 2024-08-19T14:42:22.342 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252867 2024-08-19T14:42:22.342 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-352187318275 2-317827579911 3-360777252867' 2024-08-19T14:42:22.342 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:22.342 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836537 2024-08-19T14:42:22.342 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:22.342 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:42:22.342 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836537 2024-08-19T14:42:22.342 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:22.343 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836537 2024-08-19T14:42:22.343 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-08-19T14:42:22.343 INFO:tasks.workunit.client.0.smithi102.stderr:159162: waiting osd.0 seq 21474836537 2024-08-19T14:42:22.343 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:42:22.343 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836535 -lt 21474836537 2024-08-19T14:42:22.343 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:42:22.343 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:42:22.343 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:42:22.343 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836537 2024-08-19T14:42:22.343 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:22.343 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-352187318275 2024-08-19T14:42:22.343 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:22.343 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:42:22.344 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-352187318275 2024-08-19T14:42:22.344 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:22.344 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318275 2024-08-19T14:42:22.344 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 352187318275' 2024-08-19T14:42:22.344 INFO:tasks.workunit.client.0.smithi102.stderr:159162: waiting osd.1 seq 352187318275 2024-08-19T14:42:22.344 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:42:22.344 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318275 -lt 352187318275 2024-08-19T14:42:22.344 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:23.482 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579910 2024-08-19T14:42:23.482 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:23.482 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:42:23.482 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579910 2024-08-19T14:42:23.482 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:23.482 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579910 2024-08-19T14:42:23.482 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579910' 2024-08-19T14:42:23.482 INFO:tasks.workunit.client.0.smithi102.stderr:159164: waiting osd.2 seq 317827579910 2024-08-19T14:42:23.482 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:42:23.482 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579912 -lt 317827579910 2024-08-19T14:42:23.482 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:23.483 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-360777252866 2024-08-19T14:42:23.483 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:23.483 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:42:23.483 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-360777252866 2024-08-19T14:42:23.483 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:23.483 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252866 2024-08-19T14:42:23.483 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 360777252866' 2024-08-19T14:42:23.483 INFO:tasks.workunit.client.0.smithi102.stderr:159164: waiting osd.3 seq 360777252866 2024-08-19T14:42:23.483 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:42:23.483 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252867 -lt 360777252866 2024-08-19T14:42:23.483 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:42:23.483 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:42:23.483 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:42:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:42:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:42:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:42:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:42:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:42:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:42:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:42:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:42:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:42:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:42:23.596 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.c62: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579911 2024-08-19T14:42:23.596 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:23.596 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:42:23.596 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579911 2024-08-19T14:42:23.596 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:23.596 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579911 2024-08-19T14:42:23.596 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579911' 2024-08-19T14:42:23.596 INFO:tasks.workunit.client.0.smithi102.stderr:159162: waiting osd.2 seq 317827579911 2024-08-19T14:42:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:42:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579912 -lt 317827579911 2024-08-19T14:42:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-360777252867 2024-08-19T14:42:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:42:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-360777252867 2024-08-19T14:42:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252867 2024-08-19T14:42:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 360777252867' 2024-08-19T14:42:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:159162: waiting osd.3 seq 360777252867 2024-08-19T14:42:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:42:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252867 -lt 360777252867 2024-08-19T14:42:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:42:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:42:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:42:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:42:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:42:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:42:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:42:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:42:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:42:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:42:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:42:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:42:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:42:23.823 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:42:23.823 INFO:tasks.workunit.client.0.smithi102.stderr:159164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:42:23.823 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:42:23.823 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:42:23.823 INFO:tasks.workunit.client.0.smithi102.stderr:159164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:42:23.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:42:23.957 INFO:tasks.workunit.client.0.smithi102.stderr:lient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:42:23.957 INFO:tasks.workunit.client.0.smithi102.stderr:159162: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:42:23.957 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:42:23.957 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:42:23.958 INFO:tasks.workunit.client.0.smithi102.stderr:159162: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:42:23.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:42:23.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:42:23.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 159161 2024-08-19T14:42:23.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-08-19T14:42:23.959 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-08-19T14:42:23.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-08-19T14:42:23.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-08-19T14:42:23.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-08-19T14:42:23.959 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-08-19T14:42:23.959 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T14:42:23.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T14:42:23.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:42:23.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:42:24.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-08-19T14:42:24.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-08-19T14:42:24.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-08-19T14:42:24.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-08-19T14:42:24.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:42:24.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:42:24.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:42:24.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:42:24.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T14:41:53.739238+0000 2024-08-19T14:42:24.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-08-19T14:42:24.765 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-08-19T14:42:24.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-08-19T14:41:53.739238+0000 2024-08-19T14:42:24.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T14:42:24.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:41:53.739238+0000 2024-08-19T14:42:24.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:42:24.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:42:24.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:42:24.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:42:24.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:42:24.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:42:24.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:42:24.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:42:25.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:41:53.739238+0000 '>' 2024-08-19T14:41:53.739238+0000 2024-08-19T14:42:25.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:42:26.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:42:26.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:42:26.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:42:26.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:42:26.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:42:26.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:42:26.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:42:26.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:41:53.739238+0000 '>' 2024-08-19T14:41:53.739238+0000 2024-08-19T14:42:26.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:42:27.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:42:27.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:42:27.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:42:27.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:42:27.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:42:27.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:42:27.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:42:27.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:41:53.739238+0000 '>' 2024-08-19T14:41:53.739238+0000 2024-08-19T14:42:27.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:42:28.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:42:28.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:42:28.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:42:28.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:42:28.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:42:28.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:42:28.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:42:28.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:41:53.739238+0000 '>' 2024-08-19T14:41:53.739238+0000 2024-08-19T14:42:28.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:42:29.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:42:29.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:42:29.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:42:29.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:42:29.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:42:29.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:42:29.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:42:30.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:42:25.209958+0000 '>' 2024-08-19T14:41:53.739238+0000 2024-08-19T14:42:30.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:42:30.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-08-19T14:42:30.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:42:30.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:42:30.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:42:30.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 162011"' 2024-08-19T14:42:30.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 162011' 2024-08-19T14:42:30.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:42:30.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:42:30.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:42:30.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:42:30.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 162013"' 2024-08-19T14:42:30.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 162013' 2024-08-19T14:42:30.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-08-19T14:42:30.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/162014: /' 2024-08-19T14:42:30.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 162011 162013' 2024-08-19T14:42:30.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-08-19T14:42:30.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:42:30.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 162011 2024-08-19T14:42:30.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:42:30.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/162016: /' 2024-08-19T14:42:32.669 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:42:32.669 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:42:32.669 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-08-19T14:42:32.669 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:42:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:42:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:42:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:42:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-08-19T14:42:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:42:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-08-19T14:42:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:42:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:42:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:42:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:42:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:42:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:42:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:42:32.671 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:42:32.671 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:42:32.671 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-08-19T14:42:32.671 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:42:32.671 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:42:32.671 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-08-19T14:42:32.671 INFO:tasks.workunit.client.0.smithi102.stderr:162014: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-08-19T14:42:32.671 INFO:tasks.workunit.client.0.smithi102.stderr:162014: _ 2024-08-19T14:42:32.671 INFO:tasks.workunit.client.0.smithi102.stderr:162014: hinfo_key 2024-08-19T14:42:32.671 INFO:tasks.workunit.client.0.smithi102.stderr:162014: snapset 2024-08-19T14:42:32.671 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-08-19T14:42:32.671 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:42:32.672 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:42:32.672 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-19T14:42:32.672 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:42:32.672 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:42:32.672 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:42:32.672 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:42:32.672 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:42:32.672 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:42:32.672 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T14:42:32.673 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T14:42:32.673 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:42:32.673 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:42:32.673 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:42:32.673 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:42:32.673 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:42:32.673 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:42:32.673 INFO:tasks.workunit.client.0.smithi102.stderr:162014: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:42:32.673 INFO:tasks.workunit.client.0.smithi102.stderr:162014: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:42:32.673 INFO:tasks.workunit.client.0.smithi102.stderr:162014: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:42:32.673 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:42:32.673 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:42:32.673 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:42:32.673 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:42:32.674 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:42:32.674 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:42:32.674 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:42:32.674 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:42:32.674 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:42:32.674 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:42:32.674 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:42:32.674 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T14:42:32.674 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-19T14:42:32.674 INFO:tasks.workunit.client.0.smithi102.stderr:162014: start osd.3 2024-08-19T14:42:33.019 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:42:33.019 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:42:33.019 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T14:42:33.019 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:42:33.020 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:42:33.020 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:42:33.020 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:42:33.020 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T14:42:33.020 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:42:33.020 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T14:42:33.020 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:42:33.021 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:42:33.021 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:42:33.021 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:42:33.021 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:42:33.021 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:42:33.021 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:42:33.021 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:42:33.021 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:42:33.021 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T14:42:33.021 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:42:33.021 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:42:33.022 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-08-19T14:42:33.022 INFO:tasks.workunit.client.0.smithi102.stderr:162016: _ 2024-08-19T14:42:33.022 INFO:tasks.workunit.client.0.smithi102.stderr:162016: hinfo_key 2024-08-19T14:42:33.022 INFO:tasks.workunit.client.0.smithi102.stderr:162016: snapset 2024-08-19T14:42:33.022 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T14:42:33.022 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:42:33.022 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:42:33.022 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:42:33.022 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:42:33.022 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:42:33.022 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:42:33.022 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:42:33.023 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:42:33.023 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:42:33.023 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:42:33.023 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:42:33.023 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:42:33.023 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:42:33.023 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:42:33.023 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:42:33.023 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:42:33.024 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:42:33.024 INFO:tasks.workunit.client.0.smithi102.stderr:162016: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:42:33.024 INFO:tasks.workunit.client.0.smithi102.stderr:162016: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:42:33.024 INFO:tasks.workunit.client.0.smithi102.stderr:162016: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:42:33.024 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:42:33.024 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:42:33.024 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:42:33.024 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:42:33.024 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:42:33.024 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:42:33.025 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:42:33.025 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:42:33.025 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:42:33.026 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:42:33.026 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:42:33.026 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:42:33.026 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:42:33.026 INFO:tasks.workunit.client.0.smithi102.stderr:162016: start osd.1 2024-08-19T14:42:36.797 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:42:36.797 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-08-19T14:42:36.797 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-19T14:42:36.797 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:42:36.797 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:42:36.798 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:42:36.798 INFO:tasks.workunit.client.0.smithi102.stderr:162014: 2024-08-19T14:42:32.700+0000 7f5d5801e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:36.798 INFO:tasks.workunit.client.0.smithi102.stderr:162014: 2024-08-19T14:42:32.702+0000 7f5d5801e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:36.798 INFO:tasks.workunit.client.0.smithi102.stderr:162014: 2024-08-19T14:42:32.703+0000 7f5d5801e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:36.798 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-19T14:42:36.798 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:42:36.799 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T14:42:36.799 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:42:36.799 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:42:36.799 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:36.799 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:42:36.800 INFO:tasks.workunit.client.0.smithi102.stderr:162014: 0 2024-08-19T14:42:36.800 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:36.800 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:42:36.800 INFO:tasks.workunit.client.0.smithi102.stderr:162014: 2024-08-19T14:42:33.218+0000 7f5d5801e640 -1 Falling back to public interface 2024-08-19T14:42:36.800 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:36.801 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:36.801 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:36.801 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:42:36.801 INFO:tasks.workunit.client.0.smithi102.stderr:162014: 1 2024-08-19T14:42:36.801 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:36.801 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:42:36.802 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:36.802 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:36.802 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:36.802 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:42:36.802 INFO:tasks.workunit.client.0.smithi102.stderr:162014: 2 2024-08-19T14:42:36.804 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:36.804 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:42:36.804 INFO:tasks.workunit.client.0.smithi102.stderr:162014: 2024-08-19T14:42:35.757+0000 7f5d5801e640 -1 osd.3 86 log_to_monitors true 2024-08-19T14:42:36.804 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:36.804 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:36.804 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:36.804 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:42:36.804 INFO:tasks.workunit.client.0.smithi102.stderr:162014: 3 2024-08-19T14:42:36.804 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:37.581 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-08-19T14:42:37.582 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T14:42:37.582 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:42:37.582 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:42:37.582 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:42:37.582 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:42:37.582 INFO:tasks.workunit.client.0.smithi102.stderr:162016: 2024-08-19T14:42:33.050+0000 7f897da11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:37.582 INFO:tasks.workunit.client.0.smithi102.stderr:162016: 2024-08-19T14:42:33.077+0000 7f897da11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:37.582 INFO:tasks.workunit.client.0.smithi102.stderr:162016: 2024-08-19T14:42:33.078+0000 7f897da11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:37.583 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:42:37.583 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:42:37.583 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:42:37.583 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:42:37.583 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:42:37.583 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:37.583 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:42:37.583 INFO:tasks.workunit.client.0.smithi102.stderr:162016: 0 2024-08-19T14:42:37.583 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:37.584 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:42:37.584 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:37.584 INFO:tasks.workunit.client.0.smithi102.stderr:162016: 2024-08-19T14:42:33.602+0000 7f897da11640 -1 Falling back to public interface 2024-08-19T14:42:37.584 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:37.584 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:37.584 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:42:37.584 INFO:tasks.workunit.client.0.smithi102.stderr:162016: 1 2024-08-19T14:42:37.584 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:37.584 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:42:37.584 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:37.585 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:37.585 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:37.585 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:42:37.585 INFO:tasks.workunit.client.0.smithi102.stderr:162016: 2 2024-08-19T14:42:37.585 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:37.585 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:42:37.585 INFO:tasks.workunit.client.0.smithi102.stderr:162016: 2024-08-19T14:42:36.256+0000 7f897da11640 -1 osd.1 86 log_to_monitors true 2024-08-19T14:42:37.585 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:37.585 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:37.586 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:37.586 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:42:37.586 INFO:tasks.workunit.client.0.smithi102.stderr:162016: 3 2024-08-19T14:42:37.586 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:37.586 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:42:37.956 INFO:tasks.workunit.client.0.smithi102.stderr:162016: osd.1 up in weight 1 up_from 94 up_thru 82 down_as.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:42:37.957 INFO:tasks.workunit.client.0.smithi102.stderr:162014: osd.3 up in weight 1 up_from 91 up_thru 91 down_at 87 last_clean_interval [84,86) [v2:127.0.0.1:6818/3153677357,v1:127.0.0.1:6819/3153677357] [v2:127.0.0.1:6820/3153677357,v1:127.0.0.1:6821/3153677357] exists,up 11030134-f10d-4d38-9e2b-e4fab1d556d5 2024-08-19T14:42:37.957 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:42:37.957 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:42:37.957 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:42:37.957 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:42:37.957 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:42:37.957 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:42:37.957 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:42:37.957 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:42:37.957 INFO:tasks.workunit.client.0.smithi102.stderr:162014: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:42:37.957 INFO:tasks.workunit.client.0.smithi102.stderr:162014: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:42:37.957 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:42:37.957 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:42:37.958 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:42:37.958 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:42:37.958 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:42:37.958 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:42:37.958 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:42:37.958 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:42:37.958 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:42:37.958 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:42:37.958 INFO:tasks.workunit.client.0.smithi102.stderr:162014: 1 2024-08-19T14:42:37.959 INFO:tasks.workunit.client.0.smithi102.stderr:162014: 2 2024-08-19T14:42:37.959 INFO:tasks.workunit.client.0.smithi102.stderr:162014: 3' 2024-08-19T14:42:37.959 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:42:37.959 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:37.959 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:42:37.959 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836542 2024-08-19T14:42:37.959 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836542 2024-08-19T14:42:37.959 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542' 2024-08-19T14:42:37.959 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:37.959 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:42:37.959 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925826 2024-08-19T14:42:37.959 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925826 2024-08-19T14:42:37.959 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-403726925826' 2024-08-19T14:42:38.507 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd it 87 last_clean_interval [82,86) [v2:127.0.0.1:6826/338963037,v1:127.0.0.1:6827/338963037] [v2:127.0.0.1:6828/338963037,v1:127.0.0.1:6829/338963037] exists,up 118c3f6f-c0bc-46e6-b0d9-b63cb0e35aeb 2024-08-19T14:42:38.507 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:42:38.507 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:42:38.507 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:42:38.507 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:42:38.507 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:42:38.507 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:42:38.507 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:42:38.507 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:42:38.507 INFO:tasks.workunit.client.0.smithi102.stderr:162016: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:42:38.507 INFO:tasks.workunit.client.0.smithi102.stderr:162016: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:42:38.508 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:42:38.508 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:42:38.508 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:42:38.508 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:42:38.508 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:42:38.508 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:42:38.508 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:42:38.508 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:42:38.508 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:42:38.508 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:42:38.508 INFO:tasks.workunit.client.0.smithi102.stderr:162016: 1 2024-08-19T14:42:38.508 INFO:tasks.workunit.client.0.smithi102.stderr:162016: 2 2024-08-19T14:42:38.508 INFO:tasks.workunit.client.0.smithi102.stderr:162016: 3' 2024-08-19T14:42:38.508 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:42:38.509 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:38.509 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:42:38.509 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836543 2024-08-19T14:42:38.509 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836543 2024-08-19T14:42:38.509 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543' 2024-08-19T14:42:38.509 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:38.509 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:42:38.509 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925827 2024-08-19T14:42:38.509 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925827 2024-08-19T14:42:38.509 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-403726925827' 2024-08-19T14:42:38.509 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:40.504 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: n $ids 2024-08-19T14:42:40.504 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:42:40.504 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579916 2024-08-19T14:42:40.505 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579916 2024-08-19T14:42:40.505 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-403726925826 2-317827579916' 2024-08-19T14:42:40.505 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:40.505 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:42:40.505 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023938 2024-08-19T14:42:40.505 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023938 2024-08-19T14:42:40.506 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-403726925826 2-317827579916 3-390842023938' 2024-08-19T14:42:40.506 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:40.506 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836542 2024-08-19T14:42:40.506 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:40.506 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:42:40.506 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836542 2024-08-19T14:42:40.507 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:40.507 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836542 2024-08-19T14:42:40.507 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836542' 2024-08-19T14:42:40.507 INFO:tasks.workunit.client.0.smithi102.stderr:162014: waiting osd.0 seq 21474836542 2024-08-19T14:42:40.507 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:42:40.507 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836542 2024-08-19T14:42:40.507 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:42:40.508 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:42:40.508 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:42:40.508 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836543 -lt 21474836542 2024-08-19T14:42:40.508 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:40.508 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-403726925826 2024-08-19T14:42:40.508 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:40.509 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:42:40.509 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-403726925826 2024-08-19T14:42:40.509 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:40.509 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925826 2024-08-19T14:42:40.509 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 403726925826' 2024-08-19T14:42:40.509 INFO:tasks.workunit.client.0.smithi102.stderr:162014: waiting osd.1 seq 403726925826 2024-08-19T14:42:40.509 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:42:40.510 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925826 2024-08-19T14:42:40.777 INFO:tasks.workunit.client.0.smithi102.stderr:162014 ceph tell osd.2 flush_pg_stats 2024-08-19T14:42:40.777 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579917 2024-08-19T14:42:40.777 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579917 2024-08-19T14:42:40.777 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-403726925827 2-317827579917' 2024-08-19T14:42:40.778 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:42:40.778 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:42:40.778 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023939 2024-08-19T14:42:40.778 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023939 2024-08-19T14:42:40.778 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-403726925827 2-317827579917 3-390842023939' 2024-08-19T14:42:40.778 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:40.778 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836543 2024-08-19T14:42:40.778 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:40.778 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:42:40.778 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836543 2024-08-19T14:42:40.778 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:40.778 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836543 2024-08-19T14:42:40.779 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-08-19T14:42:40.779 INFO:tasks.workunit.client.0.smithi102.stderr:162016: waiting osd.0 seq 21474836543 2024-08-19T14:42:40.779 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:42:40.779 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836543 2024-08-19T14:42:40.779 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:42:40.779 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:42:40.779 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:42:40.779 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836543 -lt 21474836543 2024-08-19T14:42:40.779 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:40.779 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-403726925827 2024-08-19T14:42:40.779 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:40.779 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:42:40.779 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-403726925827 2024-08-19T14:42:40.779 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:40.780 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925827 2024-08-19T14:42:40.780 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 403726925827' 2024-08-19T14:42:40.780 INFO:tasks.workunit.client.0.smithi102.stderr:162016: waiting osd.1 seq 403726925827 2024-08-19T14:42:40.780 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:42:40.780 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925827 2024-08-19T14:42:41.793 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seq: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:41.793 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579916 2024-08-19T14:42:41.793 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:41.793 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:42:41.793 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579916 2024-08-19T14:42:41.793 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:41.793 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579916 2024-08-19T14:42:41.793 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579916' 2024-08-19T14:42:41.793 INFO:tasks.workunit.client.0.smithi102.stderr:162014: waiting osd.2 seq 317827579916 2024-08-19T14:42:41.793 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:42:41.793 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579917 -lt 317827579916 2024-08-19T14:42:41.793 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:41.794 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-390842023938 2024-08-19T14:42:41.794 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:41.794 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:42:41.794 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-390842023938 2024-08-19T14:42:41.794 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:41.794 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023938 2024-08-19T14:42:41.794 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 390842023938' 2024-08-19T14:42:41.794 INFO:tasks.workunit.client.0.smithi102.stderr:162014: waiting osd.3 seq 390842023938 2024-08-19T14:42:41.794 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:42:41.794 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023939 -lt 390842023938 2024-08-19T14:42:41.794 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:42:41.794 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:42:41.795 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:42:41.795 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:42:41.795 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:42:41.795 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:42:41.795 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:42:41.795 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:42:41.795 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:42:41.795 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:42:41.795 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:42:41.795 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-08-19T14:42:42.017 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.cs 2024-08-19T14:42:42.018 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579917 2024-08-19T14:42:42.018 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:42.018 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:42:42.018 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579917 2024-08-19T14:42:42.018 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:42.018 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579917 2024-08-19T14:42:42.018 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579917' 2024-08-19T14:42:42.018 INFO:tasks.workunit.client.0.smithi102.stderr:162016: waiting osd.2 seq 317827579917 2024-08-19T14:42:42.018 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:42:42.018 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579917 -lt 317827579917 2024-08-19T14:42:42.019 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:42:42.019 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-390842023939 2024-08-19T14:42:42.019 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:42:42.019 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:42:42.019 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-390842023939 2024-08-19T14:42:42.019 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:42:42.019 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023939 2024-08-19T14:42:42.019 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 390842023939' 2024-08-19T14:42:42.019 INFO:tasks.workunit.client.0.smithi102.stderr:162016: waiting osd.3 seq 390842023939 2024-08-19T14:42:42.019 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:42:42.019 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023939 -lt 390842023939 2024-08-19T14:42:42.019 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:42:42.019 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:42:42.019 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:42:42.020 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:42:42.020 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:42:42.020 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:42:42.020 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:42:42.020 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:42:42.020 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:42:42.020 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:42:42.020 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:42:42.020 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-08-19T14:42:42.020 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:42:43.195 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:42:43.196 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:42:43.196 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:42:43.196 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-08-19T14:42:43.196 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-08-19T14:42:43.196 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:42:43.196 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-08-19T14:42:43.196 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:42:43.196 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:42:43.196 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:42:43.196 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:42:43.196 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:42:43.196 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:42:43.197 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:42:43.197 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:42:43.197 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:42:43.197 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:42:43.197 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-08-19T14:42:43.197 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:42:43.197 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:42:43.197 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:42:43.197 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-08-19T14:42:43.197 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-08-19T14:42:43.197 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T14:42:43.197 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T14:42:43.197 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T14:42:43.197 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T14:42:43.198 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T14:42:43.198 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T14:42:43.198 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T14:42:43.198 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2 2024-08-19T14:42:43.436 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalonlone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:42:43.436 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:42:43.437 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-08-19T14:42:43.437 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-08-19T14:42:43.437 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:42:43.437 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-08-19T14:42:43.437 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:42:43.437 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:42:43.437 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:42:43.437 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:42:43.437 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:42:43.437 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:42:43.437 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:42:43.437 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:42:43.437 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:42:43.438 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:42:43.438 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-08-19T14:42:43.438 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:42:43.438 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:42:43.438 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:42:43.438 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-08-19T14:42:43.438 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-08-19T14:42:43.438 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T14:42:43.438 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T14:42:43.438 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T14:42:43.438 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T14:42:43.438 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T14:42:43.438 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T14:42:43.439 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T14:42:43.439 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2 2024-08-19T14:42:43.439 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2 '!=' null 2024-08-19T14:42:43.914 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/e/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2 '!=' null 2024-08-19T14:42:43.914 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-08-19T14:42:43.914 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:42:43.915 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:42:43.915 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:42:43.915 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:42:43.915 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:42:43.915 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:42:43.915 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:42:43.915 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:42:43.915 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:42:43.915 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:42:43.915 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:42:43.915 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:42:43.915 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:42:43.915 INFO:tasks.workunit.client.0.smithi102.stderr:162014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:42:43.916 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:42:43.916 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:42:43.916 INFO:tasks.workunit.client.0.smithi102.stderr:162014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:42:43.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:42:43.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:42:43.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 162013 2024-08-19T14:42:44.154 INFO:tasks.workunit.client.0.smithi102.stderr:clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-08-19T14:42:44.155 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:42:44.155 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:42:44.155 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:42:44.155 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:42:44.155 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:42:44.155 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:42:44.155 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:42:44.155 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:42:44.155 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:42:44.155 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:42:44.155 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:42:44.155 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:42:44.155 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:42:44.156 INFO:tasks.workunit.client.0.smithi102.stderr:162016: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:42:44.156 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:42:44.156 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:42:44.156 INFO:tasks.workunit.client.0.smithi102.stderr:162016: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:42:44.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:42:44.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-08-19T14:42:44.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-08-19T14:42:44.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-08-19T14:42:44.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-08-19T14:42:44.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-08-19T14:42:44.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T14:42:44.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T14:42:44.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T14:42:44.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:42:44.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:42:44.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:42:44.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:42:44.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:42:44.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:42:44.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:42:44.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:42:44.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:42:44.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:42:44.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:42:44.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:42:44.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:42:44.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:42:44.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:42:44.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:42:44.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:42:44.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:42:44.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:42:44.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:42:44.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:42:44.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:42:44.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:42:44.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:42:44.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:42:44.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:42:44.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:42:44.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:42:44.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T14:42:44.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T14:42:44.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T14:42:44.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T14:42:44.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:42:44.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:42:44.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:42:44.340 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:42:44.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:42:44.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:42:44.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:42:44.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:42:44.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:42:44.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:42:44.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:42:44.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:42:44.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:42:44.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:42:44.344 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:42:44.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:42:44.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:42:44.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:42:44.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:42:44.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:42:44.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:42:44.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:42:44.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:42:44.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:42:44.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:42:44.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:42:44.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:42:44.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:42:44.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T14:42:44.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T14:42:44.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:42:44.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:42:44.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T14:42:44.354 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T14:42:44.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T14:42:44.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T14:42:44.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T14:42:44.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2024-08-19T14:42:44.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2024-08-19T14:42:44.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:820: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2024-08-19T14:42:44.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-08-19T14:42:44.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=true 2024-08-19T14:42:44.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-08-19T14:42:44.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-08-19T14:42:44.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T14:42:44.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T14:42:44.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T14:42:44.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T14:42:44.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T14:42:44.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-08-19T14:42:44.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T14:42:44.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:42:44.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:42:44.716 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:42:44.716 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:42:44.716 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:42:44.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:42:44.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T14:42:44.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T14:42:44.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T14:42:44.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:42:44.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:42:44.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T14:42:44.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T14:42:44.893 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:42:44.893 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:42:44.893 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:42:44.894 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:42:44.894 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:42:44.894 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:42:44.894 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:42:44.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:42:44.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T14:42:44.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T14:42:44.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:42:44.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:42:44.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T14:42:44.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T14:42:44.976 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:42:44.976 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:42:44.976 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:42:44.976 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:42:44.976 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:42:44.976 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:42:44.976 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:42:44.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:42:44.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T14:42:45.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-08-19T14:42:45.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T14:42:45.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T14:42:45.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T14:42:45.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T14:42:45.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T14:42:45.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T14:42:45.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T14:42:45.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:42:45.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:42:45.893 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:42:45.893 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:42:45.893 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:42:45.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:42:45.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:42:45.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:42:45.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-08-19T14:42:45.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-08-19T14:42:45.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-08-19T14:42:45.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:42:45.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:42:45.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T14:42:45.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:42:45.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:42:45.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:42:45.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:42:45.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:42:45.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:42:45.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:42:45.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:42:45.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:42:45.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:42:45.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:42:45.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:42:45.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:42:45.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:42:45.930 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:42:45.930 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:42:45.930 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:42:45.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:42:45.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:42:45.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:42:45.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:42:45.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:42:45.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:42:45.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:42:45.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:42:45.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:42:45.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:42:45.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:42:45.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:42:45.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:42:45.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:42:45.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:42:45.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0a98a077-5370-44f4-8e74-8a0e25b2dbbc 2024-08-19T14:42:45.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0a98a077-5370-44f4-8e74-8a0e25b2dbbc' 2024-08-19T14:42:45.933 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 0a98a077-5370-44f4-8e74-8a0e25b2dbbc 2024-08-19T14:42:45.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:42:45.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDlWcNmLjppOBAAalJ4+e6PrtTw/xemYQL9tA== 2024-08-19T14:42:45.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDlWcNmLjppOBAAalJ4+e6PrtTw/xemYQL9tA=="}' 2024-08-19T14:42:45.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0a98a077-5370-44f4-8e74-8a0e25b2dbbc -i td/osd-scrub-repair/0/new.json 2024-08-19T14:42:46.150 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:42:46.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T14:42:46.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDlWcNmLjppOBAAalJ4+e6PrtTw/xemYQL9tA== --osd-uuid 0a98a077-5370-44f4-8e74-8a0e25b2dbbc 2024-08-19T14:42:46.189 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:42:46.189+0000 7f3c09e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:46.208 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:42:46.208+0000 7f3c09e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:46.209 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:42:46.209+0000 7f3c09e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:46.210 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:42:46.209+0000 7f3c09e1d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:42:46.210 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:42:46.209+0000 7f3c09e1d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:42:46.210 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:42:46.209+0000 7f3c09e1d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:42:46.293 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:42:46.291+0000 7f3c09e1d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T14:42:51.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T14:42:51.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:42:51.607 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T14:42:51.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T14:42:51.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:42:51.958 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T14:42:51.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T14:42:51.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:42:51.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:42:51.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:42:51.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:42:51.985 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:42:51.984+0000 7f54e9014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:51.987 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:42:51.986+0000 7f54e9014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:51.988 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:42:51.987+0000 7f54e9014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:42:52.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T14:42:52.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:42:52.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:42:52.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:42:52.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:42:52.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:52.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:42:52.226 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:42:52.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:52.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:42:52.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:52.511 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:42:52.511+0000 7f54e9014640 -1 Falling back to public interface 2024-08-19T14:42:53.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:53.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:53.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:42:53.492 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:42:53.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:53.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:42:53.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:54.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:54.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:54.768 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:42:54.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:42:54.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:54.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:42:54.815 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:42:54.814+0000 7f54e9014640 -1 osd.0 0 log_to_monitors true 2024-08-19T14:42:55.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:56.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:56.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:56.049 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:42:56.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:42:56.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:56.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:42:56.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:57.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:57.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:57.350 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:42:57.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:42:57.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:57.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:42:57.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:42:58.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:42:58.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:42:58.743 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:42:58.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:42:58.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:42:58.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:42:59.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:00.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:00.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:00.018 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:43:00.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:43:00.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:00.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:43:00.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:01.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:01.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:01.313 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:43:01.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:43:01.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:01.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:43:01.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:02.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:02.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:02.597 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:43:02.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:43:02.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:02.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:43:02.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:03.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:03.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:03.872 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:43:03.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:43:03.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:03.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:43:04.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:05.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:05.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:05.151 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:43:05.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:43:05.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:05.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:43:05.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:06.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:06.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:06.521 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:43:06.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:43:06.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:06.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:43:06.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:07.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:07.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:07.925 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:43:07.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:43:07.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:07.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:43:08.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:09.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:09.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:09.489 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:43:09.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:43:09.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:09.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:43:10.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:11.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:11.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:11.059 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:43:11.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:43:11.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:11.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:43:11.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:12.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:12.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:12.761 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:43:12.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:43:12.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:12.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:43:13.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:14.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:14.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:14.106 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:43:14.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:43:14.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:14.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:43:14.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:15.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:15.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:15.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:43:15.666 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:43:15.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:15.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:43:16.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:17.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:17.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:17.310 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:43:17.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:43:17.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:17.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:43:17.589 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/31239702,v1:127.0.0.1:6803/31239702] [v2:127.0.0.1:6804/31239702,v1:127.0.0.1:6805/31239702] exists,up 0a98a077-5370-44f4-8e74-8a0e25b2dbbc 2024-08-19T14:43:17.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:43:17.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:43:17.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:43:17.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-08-19T14:43:17.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-08-19T14:43:17.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:43:17.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:43:17.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T14:43:17.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:43:17.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:43:17.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:43:17.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:43:17.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:43:17.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:43:17.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:43:17.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:43:17.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:43:17.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:43:17.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:43:17.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:43:17.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:43:17.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:43:17.600 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:43:17.600 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:43:17.600 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:43:17.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:43:17.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:43:17.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:43:17.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:43:17.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:43:17.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:43:17.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:43:17.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:43:17.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:43:17.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:43:17.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:43:17.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:43:17.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:43:17.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:43:17.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:43:17.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=49ecaa45-5329-4a2a-ac5b-16d876d456be 2024-08-19T14:43:17.603 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 49ecaa45-5329-4a2a-ac5b-16d876d456be 2024-08-19T14:43:17.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 49ecaa45-5329-4a2a-ac5b-16d876d456be' 2024-08-19T14:43:17.604 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:43:17.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAFWsNmKHRcJBAAxCDXWslX4gETst5FlCyShg== 2024-08-19T14:43:17.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAFWsNmKHRcJBAAxCDXWslX4gETst5FlCyShg=="}' 2024-08-19T14:43:17.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 49ecaa45-5329-4a2a-ac5b-16d876d456be -i td/osd-scrub-repair/1/new.json 2024-08-19T14:43:17.953 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:43:17.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T14:43:17.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAFWsNmKHRcJBAAxCDXWslX4gETst5FlCyShg== --osd-uuid 49ecaa45-5329-4a2a-ac5b-16d876d456be 2024-08-19T14:43:17.994 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:17.993+0000 7f36c7e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:43:18.038 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:18.037+0000 7f36c7e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:43:18.038 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:18.038+0000 7f36c7e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:43:18.039 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:18.038+0000 7f36c7e18640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:43:18.039 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:18.038+0000 7f36c7e18640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:43:18.039 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:18.038+0000 7f36c7e18640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:43:18.113 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:18.112+0000 7f36c7e18640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T14:43:23.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T14:43:23.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:43:23.802 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T14:43:23.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T14:43:23.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:43:24.149 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T14:43:24.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T14:43:24.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:43:24.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:43:24.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:43:24.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:43:24.176 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:24.175+0000 7f315c063640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:43:24.178 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:24.177+0000 7f315c063640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:43:24.179 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:24.178+0000 7f315c063640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:43:24.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T14:43:24.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:43:24.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:43:24.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:43:24.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:43:24.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:24.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:43:24.420 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:43:24.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:24.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:43:24.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:24.700 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:24.700+0000 7f315c063640 -1 Falling back to public interface 2024-08-19T14:43:25.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:25.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:25.685 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:43:25.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:43:25.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:25.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:43:25.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:26.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:26.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:26.962 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:43:26.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:43:26.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:26.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:43:26.970 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:26.968+0000 7f315c063640 -1 osd.1 0 log_to_monitors true 2024-08-19T14:43:27.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:28.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:28.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:28.241 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:43:28.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:43:28.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:28.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:43:28.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:29.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:29.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:29.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:43:29.571 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:43:29.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:29.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:43:30.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:31.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:31.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:31.015 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:43:31.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:43:31.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:31.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:43:31.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:32.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:32.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:32.295 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:43:32.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:43:32.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:32.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:43:32.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:33.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:33.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:33.599 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:43:33.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:43:33.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:33.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:43:33.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:34.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:34.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:34.882 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:43:34.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:43:34.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:34.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:43:35.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:36.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:36.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:36.164 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:43:36.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:43:36.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:36.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:43:36.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:37.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:37.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:37.532 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:43:37.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:43:37.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:37.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:43:37.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:38.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:38.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:38.925 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:43:38.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:43:38.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:38.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:43:39.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:40.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:40.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:40.302 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:43:40.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:43:40.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:40.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:43:41.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:42.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:42.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:42.116 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:43:42.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:43:42.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:42.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:43:42.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:43.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:43.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:43.720 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:43:43.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:43:43.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:43.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:43:44.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:45.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:45.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:45.220 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:43:45.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:43:45.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:45.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:43:46.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:47.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:47.007 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:43:47.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:47.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:43:47.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:47.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:43:47.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:48.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:48.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:48.569 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:43:48.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:43:48.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:48.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:43:49.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:50.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:50.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:50.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:43:50.054 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:43:50.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:50.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:43:50.369 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3409690286,v1:127.0.0.1:6811/3409690286] [v2:127.0.0.1:6812/3409690286,v1:127.0.0.1:6813/3409690286] exists,up 49ecaa45-5329-4a2a-ac5b-16d876d456be 2024-08-19T14:43:50.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:43:50.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:43:50.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:43:50.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-08-19T14:43:50.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-08-19T14:43:50.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:43:50.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:43:50.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T14:43:50.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:43:50.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T14:43:50.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:43:50.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:43:50.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:43:50.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:43:50.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T14:43:50.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T14:43:50.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:43:50.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:43:50.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:43:50.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:43:50.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:43:50.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:43:50.375 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:43:50.375 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:43:50.375 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:43:50.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:43:50.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:43:50.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:43:50.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:43:50.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:43:50.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:43:50.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:43:50.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:43:50.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:43:50.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:43:50.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:43:50.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:43:50.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:43:50.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T14:43:50.377 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:43:50.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=23ffee92-771a-48be-9a5e-0328e959a8f1 2024-08-19T14:43:50.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 23ffee92-771a-48be-9a5e-0328e959a8f1' 2024-08-19T14:43:50.378 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 23ffee92-771a-48be-9a5e-0328e959a8f1 2024-08-19T14:43:50.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:43:50.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAmWsNmMJNIFxAA1iqWiF0CgOTblrmesAcbtQ== 2024-08-19T14:43:50.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAmWsNmMJNIFxAA1iqWiF0CgOTblrmesAcbtQ=="}' 2024-08-19T14:43:50.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 23ffee92-771a-48be-9a5e-0328e959a8f1 -i td/osd-scrub-repair/2/new.json 2024-08-19T14:43:50.708 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:43:50.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-08-19T14:43:50.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAmWsNmMJNIFxAA1iqWiF0CgOTblrmesAcbtQ== --osd-uuid 23ffee92-771a-48be-9a5e-0328e959a8f1 2024-08-19T14:43:50.749 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:50.748+0000 7fb55f5b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:43:50.792 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:50.792+0000 7fb55f5b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:43:50.793 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:50.793+0000 7fb55f5b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:43:50.794 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:50.793+0000 7fb55f5b1640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:43:50.794 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:50.793+0000 7fb55f5b1640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:43:50.794 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:50.793+0000 7fb55f5b1640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:43:50.867 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:50.867+0000 7fb55f5b1640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-08-19T14:43:56.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-08-19T14:43:56.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:43:56.326 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T14:43:56.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T14:43:56.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:43:56.676 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T14:43:56.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T14:43:56.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:43:56.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:43:56.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:43:56.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:43:56.703 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:56.703+0000 7ff96552a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:43:56.705 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:56.705+0000 7ff96552a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:43:56.706 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:56.706+0000 7ff96552a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:43:56.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T14:43:56.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:43:56.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T14:43:56.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:43:56.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:43:56.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:56.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:43:56.945 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:43:56.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:56.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:43:57.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:57.230 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:57.229+0000 7ff96552a640 -1 Falling back to public interface 2024-08-19T14:43:58.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:58.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:58.214 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:43:58.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:43:58.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:58.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:43:58.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:43:59.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:43:59.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:43:59.483 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:43:59.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:43:59.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:43:59.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:43:59.558 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:43:59.557+0000 7ff96552a640 -1 osd.2 0 log_to_monitors true 2024-08-19T14:43:59.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:00.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:00.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:00.758 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:44:00.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:44:00.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:00.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:44:01.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:02.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:02.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:02.034 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:44:02.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:44:02.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:02.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:44:02.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:03.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:03.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:03.343 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:44:03.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:44:03.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:03.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:44:03.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:04.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:04.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:04.620 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:44:04.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:44:04.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:04.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:44:04.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:05.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:05.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:05.920 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:44:05.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:44:05.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:05.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:44:06.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:07.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:07.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:07.198 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:44:07.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:44:07.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:07.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:44:07.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:08.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:08.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:08.547 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:44:08.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:44:08.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:08.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:44:08.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:09.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:09.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:09.828 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:44:09.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:44:09.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:09.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:44:10.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:11.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:11.174 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:44:11.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:11.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:44:11.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:11.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:44:11.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:12.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:12.510 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:44:12.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:12.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:44:12.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:12.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:44:13.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:14.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:14.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:14.007 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:44:14.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:44:14.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:14.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:44:14.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:15.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:15.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:15.838 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:44:15.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:44:15.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:15.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:44:16.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:17.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:17.591 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:44:17.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:17.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:44:17.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:17.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:44:18.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:19.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:19.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:19.010 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:44:19.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:44:19.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:19.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:44:19.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:20.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:20.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:20.883 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:44:20.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:44:20.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:20.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:44:21.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:22.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:22.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:22.626 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:44:22.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:44:22.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:22.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:44:22.938 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1213641662,v1:127.0.0.1:6819/1213641662] [v2:127.0.0.1:6820/1213641662,v1:127.0.0.1:6821/1213641662] exists,up 23ffee92-771a-48be-9a5e-0328e959a8f1 2024-08-19T14:44:22.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:44:22.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:44:22.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:44:22.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-08-19T14:44:22.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-08-19T14:44:22.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:44:22.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:44:22.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-08-19T14:44:22.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:44:22.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:44:22.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:44:22.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:44:22.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:44:22.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:44:22.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T14:44:22.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T14:44:22.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:44:22.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:44:22.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:44:22.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:44:22.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:44:22.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:44:22.944 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:44:22.944 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:44:22.944 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:44:22.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:44:22.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:44:22.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:44:22.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:44:22.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:44:22.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:44:22.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:44:22.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:44:22.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:44:22.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:44:22.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:44:22.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:44:22.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:44:22.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T14:44:22.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:44:22.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f2573c78-0f5b-410e-9c19-12dc1c78b131 2024-08-19T14:44:22.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 f2573c78-0f5b-410e-9c19-12dc1c78b131' 2024-08-19T14:44:22.946 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 f2573c78-0f5b-410e-9c19-12dc1c78b131 2024-08-19T14:44:22.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:44:22.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGWsNmY0Q3ORAAuvN1mIwVZafHg37OI/Kusg== 2024-08-19T14:44:22.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGWsNmY0Q3ORAAuvN1mIwVZafHg37OI/Kusg=="}' 2024-08-19T14:44:22.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f2573c78-0f5b-410e-9c19-12dc1c78b131 -i td/osd-scrub-repair/3/new.json 2024-08-19T14:44:23.302 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:44:23.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-08-19T14:44:23.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGWsNmY0Q3ORAAuvN1mIwVZafHg37OI/Kusg== --osd-uuid f2573c78-0f5b-410e-9c19-12dc1c78b131 2024-08-19T14:44:23.343 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:44:23.343+0000 7f640e618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:44:23.378 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:44:23.377+0000 7f640e618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:44:23.379 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:44:23.378+0000 7f640e618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:44:23.379 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:44:23.379+0000 7f640e618640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-08-19T14:44:23.379 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:44:23.379+0000 7f640e618640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-08-19T14:44:23.380 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:44:23.379+0000 7f640e618640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-08-19T14:44:23.453 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:44:23.452+0000 7f640e618640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-08-19T14:44:29.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-08-19T14:44:29.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:44:29.247 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-08-19T14:44:29.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-08-19T14:44:29.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:44:29.587 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-08-19T14:44:29.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-08-19T14:44:29.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:44:29.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:44:29.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:44:29.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:44:29.615 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:44:29.614+0000 7f2df1bd8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:44:29.616 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:44:29.616+0000 7f2df1bd8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:44:29.618 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:44:29.617+0000 7f2df1bd8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:44:29.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-08-19T14:44:29.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:44:29.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T14:44:29.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:44:29.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:44:29.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:29.855 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:44:29.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:44:29.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:29.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:44:30.135 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:44:30.135+0000 7f2df1bd8640 -1 Falling back to public interface 2024-08-19T14:44:30.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:31.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:31.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:31.153 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:44:31.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:44:31.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:31.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:44:31.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:32.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:32.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:32.425 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:44:32.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:44:32.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:32.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:44:32.564 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:44:32.564+0000 7f2df1bd8640 -1 osd.3 0 log_to_monitors true 2024-08-19T14:44:32.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:33.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:33.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:33.775 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:44:33.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:44:33.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:33.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:44:34.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:35.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:35.049 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:44:35.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:35.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:44:35.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:35.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:44:35.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:36.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:36.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:36.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:44:36.436 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:44:36.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:36.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:44:36.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:37.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:37.835 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:44:37.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:37.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:44:37.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:37.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:44:38.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:39.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:39.112 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:44:39.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:39.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:44:39.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:39.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:44:39.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:40.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:40.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:40.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:44:40.414 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:44:40.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:40.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:44:40.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:41.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:41.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:41.687 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:44:41.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:44:41.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:41.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:44:42.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:43.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:43.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:43.015 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:44:43.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:44:43.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:43.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:44:43.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:44.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:44.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:44.386 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:44:44.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:44:44.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:44.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:44:44.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:45.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:45.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:45.663 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:44:45.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:44:45.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:45.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:44:46.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:47.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:47.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:47.228 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:44:47.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:44:47.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:47.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:44:47.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:48.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:48.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:48.531 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:44:48.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:44:48.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:48.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:44:48.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:49.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:49.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:49.826 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:44:49.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:44:49.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:49.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:44:50.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:51.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:51.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:51.167 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:44:51.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:44:51.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:51.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:44:51.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:52.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:52.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:52.937 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:44:52.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:44:52.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:52.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:44:53.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:44:54.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:44:54.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:44:54.623 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:44:54.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:44:54.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:44:54.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:44:54.913 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2142016544,v1:127.0.0.1:6827/2142016544] [v2:127.0.0.1:6828/2142016544,v1:127.0.0.1:6829/2142016544] exists,up f2573c78-0f5b-410e-9c19-12dc1c78b131 2024-08-19T14:44:54.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:44:54.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:44:54.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:44:54.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-08-19T14:44:54.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T14:44:55.174 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-08-19T14:44:55.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T14:44:55.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T14:44:55.543 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' created 2024-08-19T14:44:55.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T14:44:56.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T14:44:58.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-08-19T14:44:58.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:44:58.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:44:58.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:44:58.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:44:58.527 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:44:58.527 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:44:58.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:44:58.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:44:58.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:44:58.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:44:58.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:44:58.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:44:58.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:44:58.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:44:58.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:44:58.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:44:58.905 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:44:58.905 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:44:58.905 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-08-19T14:44:58.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:44:58.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:44:58.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:44:59.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-08-19T14:44:59.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-08-19T14:44:59.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-08-19T14:44:59.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:44:59.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:44:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-08-19T14:44:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-08-19T14:44:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672977' 2024-08-19T14:44:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:44:59.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:44:59.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-08-19T14:44:59.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-08-19T14:44:59.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672977 2-64424509456' 2024-08-19T14:44:59.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:44:59.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:44:59.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345926 2024-08-19T14:44:59.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345926 2024-08-19T14:44:59.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672977 2-64424509456 3-85899345926' 2024-08-19T14:44:59.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:44:59.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-08-19T14:44:59.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:44:59.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:44:59.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-08-19T14:44:59.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:44:59.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-08-19T14:44:59.319 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836508 2024-08-19T14:44:59.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-08-19T14:44:59.319 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:44:59.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836508 2024-08-19T14:44:59.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:45:00.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:45:00.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:45:00.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-08-19T14:45:00.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:45:00.849 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-08-19T14:45:00.849 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:45:00.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:45:00.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-08-19T14:45:00.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:45:00.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-08-19T14:45:00.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-08-19T14:45:00.852 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672977 2024-08-19T14:45:00.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:45:01.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-08-19T14:45:01.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:45:01.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-08-19T14:45:01.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:45:01.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:45:01.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-08-19T14:45:01.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:45:01.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-08-19T14:45:01.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-08-19T14:45:01.123 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509456 2024-08-19T14:45:01.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:45:01.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-08-19T14:45:01.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:45:01.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345926 2024-08-19T14:45:01.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:45:01.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:45:01.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345926 2024-08-19T14:45:01.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:45:01.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345926 2024-08-19T14:45:01.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2024-08-19T14:45:01.395 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345926 2024-08-19T14:45:01.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:45:01.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345926 -lt 85899345926 2024-08-19T14:45:01.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:45:01.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:45:01.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:45:02.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T14:45:02.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:45:02.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:45:02.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:45:02.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:45:02.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:45:02.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:45:02.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:45:02.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:45:02.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:45:02.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:45:02.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:45:02.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T14:45:02.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:45:02.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:45:02.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2024-08-19T14:45:02.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-08-19T14:45:02.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-08-19T14:45:02.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-08-19T14:45:02.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-08-19T14:45:02.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-08-19T14:45:03.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-08-19T14:45:03.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-08-19T14:45:03.606 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'ecpool' created 2024-08-19T14:45:03.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T14:45:04.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-08-19T14:45:04.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-08-19T14:45:05.297 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 2 allow_ec_overwrites to true 2024-08-19T14:45:05.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-08-19T14:45:05.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:45:05.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:45:05.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:45:05.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:45:05.310 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:45:05.310 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:45:05.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:45:05.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:45:05.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:45:05.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:45:05.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:45:05.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:45:05.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:45:05.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:45:05.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:45:05.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:45:05.688 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:45:05.688 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:45:05.688 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-08-19T14:45:05.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:45:05.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:45:05.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:45:05.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836510 2024-08-19T14:45:05.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836510 2024-08-19T14:45:05.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510' 2024-08-19T14:45:05.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:45:05.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:45:05.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-08-19T14:45:05.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-08-19T14:45:05.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672980' 2024-08-19T14:45:05.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:45:05.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:45:06.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-08-19T14:45:06.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-08-19T14:45:06.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672980 2-64424509458' 2024-08-19T14:45:06.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:45:06.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:45:06.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345929 2024-08-19T14:45:06.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345929 2024-08-19T14:45:06.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672980 2-64424509458 3-85899345929' 2024-08-19T14:45:06.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:45:06.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836510 2024-08-19T14:45:06.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:45:06.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:45:06.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836510 2024-08-19T14:45:06.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:45:06.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836510 2024-08-19T14:45:06.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-08-19T14:45:06.115 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836510 2024-08-19T14:45:06.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:45:06.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836510 2024-08-19T14:45:06.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:45:06.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-08-19T14:45:06.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:45:06.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:45:06.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-08-19T14:45:06.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:45:06.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-08-19T14:45:06.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-08-19T14:45:06.386 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672980 2024-08-19T14:45:06.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:45:06.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672980 2024-08-19T14:45:06.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:45:07.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:45:07.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:45:07.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672980 2024-08-19T14:45:07.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:45:08.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T14:45:08.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:45:09.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-08-19T14:45:09.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:45:09.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-08-19T14:45:09.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:45:09.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:45:09.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-08-19T14:45:09.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:45:09.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-08-19T14:45:09.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-08-19T14:45:09.208 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509458 2024-08-19T14:45:09.208 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:45:09.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-08-19T14:45:09.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:45:09.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345929 2024-08-19T14:45:09.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:45:09.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:45:09.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345929 2024-08-19T14:45:09.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:45:09.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345929 2024-08-19T14:45:09.479 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345929 2024-08-19T14:45:09.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2024-08-19T14:45:09.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:45:09.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345929 -lt 85899345929 2024-08-19T14:45:09.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:45:09.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:45:09.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:45:10.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:45:10.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:45:10.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:45:10.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:45:10.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:45:10.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:45:10.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:45:10.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:45:10.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:45:10.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:45:10.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:45:10.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:45:10.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:45:10.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:45:10.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:45:10.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-08-19T14:45:10.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-08-19T14:45:10.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-08-19T14:45:10.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-08-19T14:45:10.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-08-19T14:45:10.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T14:45:10.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T14:45:10.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-08-19T14:45:10.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T14:45:10.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T14:45:10.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T14:45:11.109 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T14:45:11.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T14:45:11.528 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T14:45:11.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T14:45:11.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T14:45:11.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-08-19T14:45:11.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-08-19T14:45:11.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-08-19T14:45:11.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:45:11.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:45:11.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:45:12.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-08-19T14:45:12.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-08-19T14:45:12.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-08-19T14:45:12.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-08-19T14:45:12.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-08-19T14:45:12.111 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:45:12.111 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-19T14:45:12.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-08-19T14:45:12.384 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:45:12.384 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:45:12.384 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-08-19T14:45:12.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-08-19T14:45:12.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-08-19T14:45:12.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-08-19T14:45:12.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-08-19T14:45:12.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-08-19T14:45:12.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-08-19T14:45:12.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-08-19T14:45:12.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-08-19T14:45:12.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-08-19T14:45:12.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:45:12.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:45:12.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:45:12.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-08-19T14:45:12.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:45:12.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:45:12.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:45:12.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:45:12.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-08-19T14:45:12.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:45:12.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-08-19T14:45:12.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:45:12.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:45:12.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:45:12.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:45:12.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:45:12.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:45:12.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:45:12.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:45:12.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:45:12.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-08-19T14:45:12.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:45:12.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:45:12.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-08-19T14:45:14.238 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-08-19T14:45:14.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-08-19T14:45:14.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:45:14.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:45:14.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-19T14:45:14.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:45:14.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:45:14.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:45:14.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:45:14.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:45:14.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:45:14.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T14:45:14.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T14:45:14.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:45:14.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:45:14.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:45:14.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:45:14.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:45:14.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:45:14.758 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:45:14.759 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:45:14.759 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:45:14.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:45:14.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:45:14.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:45:14.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:45:14.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:45:14.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:45:14.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:45:14.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:45:14.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:45:14.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:45:14.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:45:14.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T14:45:14.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-19T14:45:14.760 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.3 2024-08-19T14:45:14.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:45:14.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-08-19T14:45:14.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-19T14:45:14.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:45:14.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:45:14.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:45:14.788 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:45:14.787+0000 7fc502e82640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:45:14.790 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:45:14.789+0000 7fc502e82640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:45:14.791 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:45:14.791+0000 7fc502e82640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:45:15.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-19T14:45:15.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:45:15.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T14:45:15.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:45:15.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:45:15.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:45:15.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:45:15.040 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:45:15.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:45:15.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:45:15.312 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:45:15.312+0000 7fc502e82640 -1 Falling back to public interface 2024-08-19T14:45:15.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:45:16.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:45:16.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:45:16.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:45:16.348 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:45:16.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:45:16.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:45:16.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:45:17.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:45:17.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:45:17.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:45:17.632 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:45:17.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:45:17.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:45:17.648 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:45:17.648+0000 7fc502e82640 -1 osd.3 36 log_to_monitors true 2024-08-19T14:45:17.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:45:18.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:45:18.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:45:18.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:45:18.991 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:45:18.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:45:18.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:45:19.292 INFO:tasks.workunit.client.0.smithi102.stderr:osd.3 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [20,36) [v2:127.0.0.1:6826/2862067724,v1:127.0.0.1:6827/2862067724] [v2:127.0.0.1:6828/2862067724,v1:127.0.0.1:6829/2862067724] exists,up f2573c78-0f5b-410e-9c19-12dc1c78b131 2024-08-19T14:45:19.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:45:19.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:45:19.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:45:19.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:45:19.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:45:19.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:45:19.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:45:19.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:45:19.292 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:45:19.293 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:45:19.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:45:19.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:45:19.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:45:19.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:45:19.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:45:19.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:45:19.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:45:19.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:45:19.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:45:19.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:45:19.759 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:45:19.759 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:45:19.759 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-08-19T14:45:19.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:45:19.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:45:19.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:45:19.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836514 2024-08-19T14:45:19.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836514 2024-08-19T14:45:19.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514' 2024-08-19T14:45:19.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:45:19.861 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:45:19.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-08-19T14:45:19.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-08-19T14:45:19.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672983' 2024-08-19T14:45:19.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:45:19.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:45:20.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-08-19T14:45:20.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-08-19T14:45:20.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672983 2-64424509462' 2024-08-19T14:45:20.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:45:20.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:45:20.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-08-19T14:45:20.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-08-19T14:45:20.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672983 2-64424509462 3-171798691842' 2024-08-19T14:45:20.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:45:20.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836514 2024-08-19T14:45:20.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:45:20.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:45:20.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836514 2024-08-19T14:45:20.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:45:20.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836514 2024-08-19T14:45:20.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-08-19T14:45:20.174 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836514 2024-08-19T14:45:20.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:45:20.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836514 2024-08-19T14:45:20.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:45:20.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-08-19T14:45:20.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:45:20.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:45:20.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-08-19T14:45:20.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:45:20.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-08-19T14:45:20.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-08-19T14:45:20.444 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949672983 2024-08-19T14:45:20.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:45:20.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672983 2024-08-19T14:45:20.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:45:21.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:45:21.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:45:21.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672983 2024-08-19T14:45:21.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:45:22.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T14:45:22.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:45:23.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672983 2024-08-19T14:45:23.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:45:23.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-08-19T14:45:23.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:45:23.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:45:23.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-08-19T14:45:23.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:45:23.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-08-19T14:45:23.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-08-19T14:45:23.263 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509462 2024-08-19T14:45:23.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:45:23.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-08-19T14:45:23.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:45:23.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-171798691842 2024-08-19T14:45:23.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:45:23.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:45:23.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-171798691842 2024-08-19T14:45:23.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:45:23.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-08-19T14:45:23.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 171798691842' 2024-08-19T14:45:23.533 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.3 seq 171798691842 2024-08-19T14:45:23.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:45:23.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-08-19T14:45:23.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:45:23.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:45:23.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:45:24.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:45:24.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:45:24.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:45:24.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:45:24.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:45:24.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:45:24.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:45:24.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:45:24.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:45:24.396 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:45:24.397 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:45:24.397 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:45:24.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:45:24.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:45:24.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:45:24.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-08-19T14:45:24.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T14:45:24.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T14:45:24.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:45:24.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:45:25.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-08-19T14:45:25.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-08-19T14:45:25.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-08-19T14:45:25.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-08-19T14:45:25.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:45:25.044 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:45:25.044 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:45:25.044 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:45:25.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T14:45:03.521451+0000 2024-08-19T14:45:25.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-08-19T14:45:25.555 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-08-19T14:45:25.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-08-19T14:45:03.521451+0000 2024-08-19T14:45:25.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T14:45:25.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:45:03.521451+0000 2024-08-19T14:45:25.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:45:25.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:45:25.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:45:25.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:45:25.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:45:25.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:45:25.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:45:25.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:45:25.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:45:03.521451+0000 '>' 2024-08-19T14:45:03.521451+0000 2024-08-19T14:45:25.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:45:26.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:45:26.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:45:26.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:45:26.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:45:26.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:45:26.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:45:26.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:45:27.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:45:03.521451+0000 '>' 2024-08-19T14:45:03.521451+0000 2024-08-19T14:45:27.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:45:28.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:45:28.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:45:28.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:45:28.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:45:28.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:45:28.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:45:28.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:45:28.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:45:03.521451+0000 '>' 2024-08-19T14:45:03.521451+0000 2024-08-19T14:45:28.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:45:29.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:45:29.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:45:29.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:45:29.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:45:29.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:45:29.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:45:29.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:45:29.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:45:03.521451+0000 '>' 2024-08-19T14:45:03.521451+0000 2024-08-19T14:45:29.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:45:30.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:45:30.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:45:30.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:45:30.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:45:30.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:45:30.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:45:30.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:45:30.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:45:25.797372+0000 '>' 2024-08-19T14:45:03.521451+0000 2024-08-19T14:45:30.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:45:30.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:45:30.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:45:30.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:45:30.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-08-19T14:45:30.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:45:30.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:45:30.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:45:30.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:45:30.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-08-19T14:45:30.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:45:30.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-08-19T14:45:30.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:45:30.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:45:30.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:45:30.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:45:30.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:45:31.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:45:31.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:45:31.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:45:31.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:45:31.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-08-19T14:45:31.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:45:31.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:45:31.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-08-19T14:45:32.584 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-08-19T14:45:32.646 INFO:tasks.workunit.client.0.smithi102.stdout:_ 2024-08-19T14:45:32.646 INFO:tasks.workunit.client.0.smithi102.stdout:hinfo_key 2024-08-19T14:45:32.646 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 2024-08-19T14:45:32.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-08-19T14:45:32.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:45:32.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:45:32.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-19T14:45:32.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:45:32.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:45:32.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:45:32.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:45:32.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:45:32.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:45:32.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T14:45:32.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T14:45:32.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:45:32.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:45:32.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:45:32.861 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:45:32.861 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:45:32.861 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:45:32.861 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:45:32.861 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:45:32.862 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:45:32.862 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:45:32.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:45:32.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:45:32.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:45:32.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:45:32.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:45:32.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:45:32.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:45:32.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:45:32.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:45:32.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:45:32.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T14:45:32.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-19T14:45:32.863 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.3 2024-08-19T14:45:32.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:45:32.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-08-19T14:45:32.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-19T14:45:32.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:45:32.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:45:32.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:45:32.890 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:45:32.889+0000 7fc718301640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:45:32.893 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:45:32.892+0000 7fc718301640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:45:32.894 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:45:32.893+0000 7fc718301640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:45:33.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-19T14:45:33.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:45:33.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T14:45:33.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:45:33.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:45:33.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:45:33.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:45:33.144 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:45:33.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:45:33.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:45:33.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:45:33.419 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:45:33.419+0000 7fc718301640 -1 Falling back to public interface 2024-08-19T14:45:34.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:45:34.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:45:34.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:45:34.410 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:45:34.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:45:34.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:45:34.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:45:35.647 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:45:35.646+0000 7fc718301640 -1 osd.3 42 log_to_monitors true 2024-08-19T14:45:35.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:45:35.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:45:35.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:45:35.677 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:45:35.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:45:35.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:45:36.099 INFO:tasks.workunit.client.0.smithi102.stderr:osd.3 up in weight 1 up_from 46 up_thru 40 down_at 43 last_clean_interval [40,42) [v2:127.0.0.1:6826/3201546051,v1:127.0.0.1:6827/3201546051] [v2:127.0.0.1:6828/3201546051,v1:127.0.0.1:6829/3201546051] exists,up f2573c78-0f5b-410e-9c19-12dc1c78b131 2024-08-19T14:45:36.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:45:36.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:45:36.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:45:36.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:45:36.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:45:36.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:45:36.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:45:36.100 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:45:36.101 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:45:36.101 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:45:36.101 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:45:36.101 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:45:36.101 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:45:36.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:45:36.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:45:36.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:45:36.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:45:36.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:45:36.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:45:36.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:45:36.523 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:45:36.523 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:45:36.523 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-08-19T14:45:36.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:45:36.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:45:36.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:45:36.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836518 2024-08-19T14:45:36.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836518 2024-08-19T14:45:36.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518' 2024-08-19T14:45:36.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:45:36.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:45:36.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672988 2024-08-19T14:45:36.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672988 2024-08-19T14:45:36.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-42949672988' 2024-08-19T14:45:36.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:45:36.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:45:37.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509466 2024-08-19T14:45:37.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509466 2024-08-19T14:45:37.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-42949672988 2-64424509466' 2024-08-19T14:45:37.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:45:37.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:45:37.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495618 2024-08-19T14:45:37.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495618 2024-08-19T14:45:37.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-42949672988 2-64424509466 3-197568495618' 2024-08-19T14:45:37.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:45:37.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836518 2024-08-19T14:45:37.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:45:37.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:45:37.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836518 2024-08-19T14:45:37.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:45:37.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836518 2024-08-19T14:45:37.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-08-19T14:45:37.294 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836518 2024-08-19T14:45:37.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:45:37.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836517 -lt 21474836518 2024-08-19T14:45:37.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:45:38.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:45:38.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:45:38.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836519 -lt 21474836518 2024-08-19T14:45:38.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:45:38.834 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672988 2024-08-19T14:45:38.834 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:45:38.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:45:38.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672988 2024-08-19T14:45:38.836 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:45:38.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672988 2024-08-19T14:45:38.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-08-19T14:45:38.837 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949672988 2024-08-19T14:45:38.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:45:39.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672988 -lt 42949672988 2024-08-19T14:45:39.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:45:39.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509466 2024-08-19T14:45:39.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:45:39.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:45:39.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509466 2024-08-19T14:45:39.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:45:39.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509466 2024-08-19T14:45:39.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-08-19T14:45:39.113 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509466 2024-08-19T14:45:39.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:45:39.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509466 -lt 64424509466 2024-08-19T14:45:39.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:45:39.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-197568495618 2024-08-19T14:45:39.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:45:39.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:45:39.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-197568495618 2024-08-19T14:45:39.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:45:39.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495618 2024-08-19T14:45:39.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 197568495618' 2024-08-19T14:45:39.384 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.3 seq 197568495618 2024-08-19T14:45:39.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:45:39.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495618 -lt 197568495618 2024-08-19T14:45:39.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:45:39.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:45:39.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:45:39.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:45:39.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:45:39.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:45:39.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:45:39.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:45:39.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:45:39.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:45:39.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:45:40.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:45:40.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:45:40.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:45:40.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:45:40.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-08-19T14:45:40.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-08-19T14:45:40.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:45:40.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-08-19T14:45:40.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:45:40.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:45:40.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:45:40.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:45:40.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:45:40.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:45:40.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:45:40.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:45:40.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:45:40.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:45:40.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:45:40.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:45:40.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:45:40.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:45:41.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-08-19T14:45:41.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-08-19T14:45:41.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T14:45:41.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T14:45:41.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T14:45:41.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T14:45:41.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T14:45:41.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T14:45:41.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T14:45:41.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T14:45:41.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T14:45:41.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-19T14:45:41.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:45:41.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-19T14:45:41.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:45:41.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:45:41.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:45:41.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:45:41.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:45:41.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:45:41.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:45:41.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:45:42.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:45:42.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:45:42.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:45:42.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:45:42.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-08-19T14:45:42.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-08-19T14:45:42.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T14:45:42.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T14:45:42.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T14:45:42.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T14:45:42.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T14:45:42.397 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T14:45:42.398 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T14:45:42.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T14:45:42.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T14:45:42.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-08-19T14:45:42.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:45:42.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-19T14:45:43.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:45:43.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:45:43.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:45:43.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:45:43.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:45:43.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:45:43.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:45:43.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:45:43.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:45:43.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:45:43.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:45:43.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:45:43.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-08-19T14:45:43.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-08-19T14:45:43.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T14:45:43.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T14:45:43.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T14:45:43.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T14:45:43.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T14:45:43.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T14:45:43.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T14:45:44.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T14:45:44.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T14:45:44.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-08-19T14:45:44.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:45:44.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-08-19T14:45:44.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:45:44.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:45:44.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:45:44.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:45:44.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:45:44.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:45:44.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:45:44.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:45:45.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:45:45.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:45:45.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:45:45.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:45:45.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:45:45.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:45:45.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:45:45.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T14:45:45.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T14:45:45.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-08-19T14:45:45.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-08-19T14:45:45.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-08-19T14:45:45.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-08-19T14:45:45.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:45:45.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:45:45.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:45:45.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T14:45:45.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:45:45.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:45:45.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:45:45.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:45:45.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T14:45:45.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:45:45.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T14:45:45.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:45:45.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:45:45.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:45:45.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:45:45.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:45:45.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:45:45.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:45:45.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:45:45.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:45:45.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T14:45:45.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:45:45.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:45:45.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-08-19T14:45:47.434 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-08-19T14:45:47.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T14:45:47.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:45:47.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:45:47.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:45:47.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:45:47.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:45:47.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:45:47.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:45:47.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:45:47.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:45:47.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:45:47.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:45:47.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:45:47.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:45:47.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:45:47.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:45:47.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:45:47.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:45:47.956 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:45:47.956 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:45:47.956 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:45:47.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:45:47.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:45:47.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:45:47.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:45:47.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:45:47.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:45:47.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:45:47.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:45:47.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:45:47.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:45:47.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:45:47.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:45:47.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:45:47.958 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T14:45:47.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:45:47.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T14:45:47.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:45:47.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:45:47.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:45:47.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:45:47.985 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:45:47.985+0000 7fbeec282640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:45:47.988 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:45:47.987+0000 7fbeec282640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:45:47.989 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:45:47.989+0000 7fbeec282640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:45:48.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:45:48.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:45:48.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:45:48.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:45:48.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:45:48.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:45:48.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:45:48.240 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:45:48.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:45:48.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:45:48.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:45:48.510 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:45:48.510+0000 7fbeec282640 -1 Falling back to public interface 2024-08-19T14:45:49.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:45:49.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:45:49.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:45:49.506 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:45:49.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:45:49.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:45:49.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:45:50.764 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:45:50.763+0000 7fbeec282640 -1 osd.1 48 log_to_monitors true 2024-08-19T14:45:50.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:45:50.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:45:50.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:45:50.780 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:45:50.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:45:50.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:45:51.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:45:52.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:45:52.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:45:52.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:45:52.123 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:45:52.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:45:52.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:45:52.462 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 53 up_thru 53 down_at 49 last_clean_interval [10,48) [v2:127.0.0.1:6810/3049961597,v1:127.0.0.1:6811/3049961597] [v2:127.0.0.1:6812/3049961597,v1:127.0.0.1:6813/3049961597] exists,up 49ecaa45-5329-4a2a-ac5b-16d876d456be 2024-08-19T14:45:52.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:45:52.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:45:52.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:45:52.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:45:52.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:45:52.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:45:52.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:45:52.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:45:52.463 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:45:52.463 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:45:52.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:45:52.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:45:52.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:45:52.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:45:52.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:45:52.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:45:52.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:45:52.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:45:52.552 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:45:52.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:45:52.818 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:45:52.818 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:45:52.818 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-08-19T14:45:52.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:45:52.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:45:52.818 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:45:52.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836523 2024-08-19T14:45:52.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836523 2024-08-19T14:45:52.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523' 2024-08-19T14:45:52.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:45:52.920 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:45:53.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266690 2024-08-19T14:45:53.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266690 2024-08-19T14:45:53.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-227633266690' 2024-08-19T14:45:53.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:45:53.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:45:53.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509471 2024-08-19T14:45:53.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509471 2024-08-19T14:45:53.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-227633266690 2-64424509471' 2024-08-19T14:45:53.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:45:53.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:45:53.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495622 2024-08-19T14:45:53.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495622 2024-08-19T14:45:53.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-227633266690 2-64424509471 3-197568495622' 2024-08-19T14:45:53.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:45:53.227 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836523 2024-08-19T14:45:53.227 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:45:53.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:45:53.228 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836523 2024-08-19T14:45:53.228 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:45:53.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836523 2024-08-19T14:45:53.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2024-08-19T14:45:53.229 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836523 2024-08-19T14:45:53.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:45:53.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836521 -lt 21474836523 2024-08-19T14:45:53.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:45:54.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:45:54.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:45:54.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836523 -lt 21474836523 2024-08-19T14:45:54.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:45:54.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-227633266690 2024-08-19T14:45:54.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:45:54.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:45:54.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-227633266690 2024-08-19T14:45:54.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:45:54.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266690 2024-08-19T14:45:54.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 227633266690' 2024-08-19T14:45:54.776 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 227633266690 2024-08-19T14:45:54.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:45:55.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266690 -lt 227633266690 2024-08-19T14:45:55.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:45:55.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509471 2024-08-19T14:45:55.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:45:55.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:45:55.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509471 2024-08-19T14:45:55.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:45:55.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509471 2024-08-19T14:45:55.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-08-19T14:45:55.078 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509471 2024-08-19T14:45:55.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:45:55.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509471 -lt 64424509471 2024-08-19T14:45:55.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:45:55.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-197568495622 2024-08-19T14:45:55.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:45:55.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:45:55.388 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-197568495622 2024-08-19T14:45:55.388 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:45:55.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495622 2024-08-19T14:45:55.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 197568495622' 2024-08-19T14:45:55.389 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.3 seq 197568495622 2024-08-19T14:45:55.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:45:55.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495622 -lt 197568495622 2024-08-19T14:45:55.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:45:55.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:45:55.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:45:56.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:45:56.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:45:56.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:45:56.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:45:56.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:45:56.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:45:56.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:45:56.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:45:56.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:45:56.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:45:56.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:45:56.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:45:56.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:45:56.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:45:56.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:45:56.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-08-19T14:45:56.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T14:45:56.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T14:45:56.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:45:56.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:45:56.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-08-19T14:45:56.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-08-19T14:45:56.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-08-19T14:45:56.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-08-19T14:45:56.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:45:56.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:45:56.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:45:56.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:45:57.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T14:45:25.797372+0000 2024-08-19T14:45:57.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-08-19T14:45:57.420 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-08-19T14:45:57.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-08-19T14:45:25.797372+0000 2024-08-19T14:45:57.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T14:45:57.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:45:25.797372+0000 2024-08-19T14:45:57.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:45:57.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:45:57.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:45:57.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:45:57.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:45:57.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:45:57.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:45:57.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:45:57.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:45:25.797372+0000 '>' 2024-08-19T14:45:25.797372+0000 2024-08-19T14:45:57.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:45:58.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:45:58.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:45:58.698 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:45:58.698 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:45:58.698 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:45:58.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:45:58.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:45:58.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:45:25.797372+0000 '>' 2024-08-19T14:45:25.797372+0000 2024-08-19T14:45:58.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:45:59.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:45:59.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:45:59.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:45:59.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:45:59.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:45:59.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:45:59.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:46:00.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:45:25.797372+0000 '>' 2024-08-19T14:45:25.797372+0000 2024-08-19T14:46:00.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:46:01.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:46:01.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:46:01.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:46:01.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:46:01.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:46:01.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:46:01.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:46:01.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:45:25.797372+0000 '>' 2024-08-19T14:45:25.797372+0000 2024-08-19T14:46:01.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:46:02.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:46:02.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:46:02.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:46:02.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:46:02.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:46:02.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:46:02.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:46:02.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:45:57.477184+0000 '>' 2024-08-19T14:45:25.797372+0000 2024-08-19T14:46:02.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:46:02.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:46:02.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:46:02.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:46:02.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-08-19T14:46:02.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:46:02.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:46:02.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:46:02.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:46:02.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-08-19T14:46:02.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:46:02.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-08-19T14:46:02.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:46:02.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:46:02.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:46:02.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:46:02.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:46:03.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:46:03.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:46:03.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:46:03.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:46:03.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-08-19T14:46:03.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:46:03.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:46:03.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-08-19T14:46:04.413 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-08-19T14:46:04.413 INFO:tasks.workunit.client.0.smithi102.stdout:_ 2024-08-19T14:46:04.413 INFO:tasks.workunit.client.0.smithi102.stdout:hinfo_key 2024-08-19T14:46:04.413 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 2024-08-19T14:46:04.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-08-19T14:46:04.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:46:04.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:46:04.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-19T14:46:04.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:46:04.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:46:04.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:46:04.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:46:04.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:46:04.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:46:04.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T14:46:04.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T14:46:04.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:46:04.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:46:04.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:46:04.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:46:04.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:46:04.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:46:04.688 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:46:04.688 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:46:04.688 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:46:04.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:46:04.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:46:04.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:46:04.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:46:04.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:46:04.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:46:04.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:46:04.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:46:04.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:46:04.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:46:04.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:46:04.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T14:46:04.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-19T14:46:04.690 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.3 2024-08-19T14:46:04.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:46:04.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-08-19T14:46:04.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-19T14:46:04.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:46:04.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:46:04.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:46:04.717 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:46:04.717+0000 7fa11de1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:04.720 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:46:04.719+0000 7fa11de1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:04.721 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:46:04.720+0000 7fa11de1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:04.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-19T14:46:04.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:46:04.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T14:46:04.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:46:04.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:46:04.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:04.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:46:04.971 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:46:04.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:04.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:46:05.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:05.243 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:46:05.243+0000 7fa11de1e640 -1 Falling back to public interface 2024-08-19T14:46:06.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:06.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:06.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:46:06.237 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:46:06.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:06.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:46:06.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:07.420 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:46:07.420+0000 7fa11de1e640 -1 osd.3 55 log_to_monitors true 2024-08-19T14:46:07.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:07.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:07.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:46:07.509 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:46:07.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:07.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:46:07.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:08.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:08.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:08.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:46:08.792 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:46:08.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:08.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:46:09.069 INFO:tasks.workunit.client.0.smithi102.stderr:osd.3 up in weight 1 up_from 59 up_thru 59 down_at 56 last_clean_interval [46,55) [v2:127.0.0.1:6826/4029925867,v1:127.0.0.1:6827/4029925867] [v2:127.0.0.1:6828/4029925867,v1:127.0.0.1:6829/4029925867] exists,up f2573c78-0f5b-410e-9c19-12dc1c78b131 2024-08-19T14:46:09.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:46:09.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:46:09.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:46:09.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:46:09.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:46:09.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:46:09.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:46:09.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:46:09.070 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:46:09.070 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:46:09.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:46:09.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:46:09.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:46:09.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:46:09.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:46:09.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:46:09.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:46:09.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:46:09.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:46:09.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:46:09.425 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:46:09.425 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:46:09.425 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-08-19T14:46:09.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:46:09.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:09.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:46:09.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836527 2024-08-19T14:46:09.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836527 2024-08-19T14:46:09.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527' 2024-08-19T14:46:09.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:09.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:46:09.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266694 2024-08-19T14:46:09.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266694 2024-08-19T14:46:09.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-227633266694' 2024-08-19T14:46:09.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:09.631 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:46:09.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509475 2024-08-19T14:46:09.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509475 2024-08-19T14:46:09.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-227633266694 2-64424509475' 2024-08-19T14:46:09.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:09.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:46:09.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070466 2024-08-19T14:46:09.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070466 2024-08-19T14:46:09.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-227633266694 2-64424509475 3-253403070466' 2024-08-19T14:46:09.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:09.836 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836527 2024-08-19T14:46:09.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:09.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:46:09.838 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836527 2024-08-19T14:46:09.838 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:09.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836527 2024-08-19T14:46:09.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-08-19T14:46:09.839 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836527 2024-08-19T14:46:09.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:46:10.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836527 -lt 21474836527 2024-08-19T14:46:10.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:10.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-227633266694 2024-08-19T14:46:10.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:10.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:46:10.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-227633266694 2024-08-19T14:46:10.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:10.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266694 2024-08-19T14:46:10.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 227633266694' 2024-08-19T14:46:10.120 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 227633266694 2024-08-19T14:46:10.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:46:10.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266694 -lt 227633266694 2024-08-19T14:46:10.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:10.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509475 2024-08-19T14:46:10.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:10.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:46:10.388 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509475 2024-08-19T14:46:10.388 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:10.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509475 2024-08-19T14:46:10.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2024-08-19T14:46:10.389 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509475 2024-08-19T14:46:10.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:46:10.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509475 -lt 64424509475 2024-08-19T14:46:10.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:10.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070466 2024-08-19T14:46:10.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:10.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:46:10.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070466 2024-08-19T14:46:10.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:10.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070466 2024-08-19T14:46:10.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070466' 2024-08-19T14:46:10.662 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.3 seq 253403070466 2024-08-19T14:46:10.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:46:10.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070466 -lt 253403070466 2024-08-19T14:46:10.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:46:10.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:46:10.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:46:11.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:46:11.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:46:11.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:46:11.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:46:11.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:46:11.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:46:11.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:46:11.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:46:11.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:46:11.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:46:11.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:46:11.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:46:11.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:46:11.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:46:11.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:46:11.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T14:46:11.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T14:46:11.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-08-19T14:46:11.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-08-19T14:46:11.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-08-19T14:46:11.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-08-19T14:46:11.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-08-19T14:46:11.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-08-19T14:46:11.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:46:11.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:46:11.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:46:11.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 179170"' 2024-08-19T14:46:11.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 179170' 2024-08-19T14:46:11.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-08-19T14:46:11.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:46:11.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:46:11.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 179172"' 2024-08-19T14:46:11.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 179172' 2024-08-19T14:46:11.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:46:11.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-08-19T14:46:11.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 179170 179172' 2024-08-19T14:46:11.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-08-19T14:46:11.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:46:11.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 179170 2024-08-19T14:46:11.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/179173: /' 2024-08-19T14:46:11.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-08-19T14:46:11.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/179175: /' 2024-08-19T14:46:14.715 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:46:14.715 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:46:14.715 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T14:46:14.715 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:46:14.715 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:46:14.716 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:46:14.716 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:46:14.716 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T14:46:14.716 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:46:14.716 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T14:46:14.716 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:46:14.716 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:46:14.716 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:46:14.716 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:46:14.716 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:46:14.716 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:46:14.716 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:46:14.717 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:46:14.717 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:46:14.717 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T14:46:14.717 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:46:14.717 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:46:14.717 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-08-19T14:46:14.717 INFO:tasks.workunit.client.0.smithi102.stderr:179173: remove 1#2:eb822e21:::SOMETHING:head# 2024-08-19T14:46:14.717 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T14:46:14.717 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:46:14.717 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:46:14.717 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:46:14.717 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:46:14.717 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:46:14.717 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:46:14.718 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:46:14.718 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:46:14.719 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:46:14.719 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:46:14.719 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:46:14.719 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:46:14.719 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:46:14.719 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:46:14.719 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:46:14.719 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:46:14.719 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:46:14.719 INFO:tasks.workunit.client.0.smithi102.stderr:179173: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:46:14.719 INFO:tasks.workunit.client.0.smithi102.stderr:179173: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:46:14.719 INFO:tasks.workunit.client.0.smithi102.stderr:179173: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:46:14.720 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:46:14.720 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:46:14.720 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:46:14.720 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:46:14.720 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:46:14.720 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:46:14.720 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:46:14.720 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:46:14.720 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:46:14.720 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:46:14.720 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:46:14.720 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:46:14.721 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:46:14.721 INFO:tasks.workunit.client.0.smithi102.stderr:179173: start osd.1 2024-08-19T14:46:14.721 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:46:14.836 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:46:14.837 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:46:14.837 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-08-19T14:46:14.837 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:46:14.837 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-08-19T14:46:14.837 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:46:14.837 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:46:14.837 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-08-19T14:46:14.837 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:46:14.837 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-08-19T14:46:14.837 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:46:14.837 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:46:14.837 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:46:14.837 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:46:14.838 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:46:14.838 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:46:14.838 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-08-19T14:46:14.838 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:46:14.838 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:46:14.838 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-08-19T14:46:14.838 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:46:14.838 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-08-19T14:46:14.838 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-08-19T14:46:14.838 INFO:tasks.workunit.client.0.smithi102.stderr:179175: remove 2#2:eb822e21:::SOMETHING:head# 2024-08-19T14:46:14.838 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-08-19T14:46:14.838 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:46:14.839 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:46:14.839 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-19T14:46:14.839 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:46:14.839 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T14:46:14.839 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:46:14.839 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:46:14.839 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:46:14.839 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:46:14.839 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T14:46:14.839 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T14:46:14.840 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:46:14.840 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:46:14.840 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:46:14.840 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:46:14.840 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:46:14.840 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:46:14.840 INFO:tasks.workunit.client.0.smithi102.stderr:179175: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:46:14.840 INFO:tasks.workunit.client.0.smithi102.stderr:179175: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:46:14.840 INFO:tasks.workunit.client.0.smithi102.stderr:179175: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:46:14.840 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:46:14.840 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:46:14.840 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:46:14.840 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:46:14.841 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:46:14.841 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:46:14.841 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:46:14.841 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:46:14.841 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:46:14.841 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:46:14.841 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:46:14.841 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T14:46:14.841 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-19T14:46:14.841 INFO:tasks.workunit.client.0.smithi102.stderr:179175: start osd.2 2024-08-19T14:46:14.841 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:46:19.202 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T14:46:19.202 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:46:19.202 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:46:19.203 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:46:19.203 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:46:19.203 INFO:tasks.workunit.client.0.smithi102.stderr:179173: 2024-08-19T14:46:14.747+0000 7ff398e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:19.203 INFO:tasks.workunit.client.0.smithi102.stderr:179173: 2024-08-19T14:46:14.749+0000 7ff398e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:19.203 INFO:tasks.workunit.client.0.smithi102.stderr:179173: 2024-08-19T14:46:14.750+0000 7ff398e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:19.204 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:46:19.204 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:46:19.204 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:46:19.204 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:46:19.204 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:46:19.205 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:19.205 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:46:19.205 INFO:tasks.workunit.client.0.smithi102.stderr:179173: 0 2024-08-19T14:46:19.205 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:19.205 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:46:19.205 INFO:tasks.workunit.client.0.smithi102.stderr:179173: 2024-08-19T14:46:15.272+0000 7ff398e1d640 -1 Falling back to public interface 2024-08-19T14:46:19.206 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:19.206 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:19.206 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:19.206 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:46:19.206 INFO:tasks.workunit.client.0.smithi102.stderr:179173: 1 2024-08-19T14:46:19.207 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:19.207 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:46:19.207 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:19.207 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:19.207 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:19.208 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:46:19.208 INFO:tasks.workunit.client.0.smithi102.stderr:179173: 2 2024-08-19T14:46:19.208 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:19.208 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:46:19.208 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:19.208 INFO:tasks.workunit.client.0.smithi102.stderr:179173: 2024-08-19T14:46:17.892+0000 7ff398e1d640 -1 osd.1 60 log_to_monitors true 2024-08-19T14:46:19.209 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:19.209 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:19.209 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:46:19.209 INFO:tasks.workunit.client.0.smithi102.stderr:179173: 3 2024-08-19T14:46:19.209 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:19.210 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:46:19.418 INFO:tasks.workunit.client.0.smithi102.stderr:179173: osd.1 up in weight 1 up_from 66 up_thru 66 down_at 61 last_179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-08-19T14:46:19.419 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-19T14:46:19.419 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:46:19.419 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:46:19.419 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:46:19.419 INFO:tasks.workunit.client.0.smithi102.stderr:179175: 2024-08-19T14:46:14.865+0000 7f24c65c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:19.419 INFO:tasks.workunit.client.0.smithi102.stderr:179175: 2024-08-19T14:46:14.868+0000 7f24c65c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:19.419 INFO:tasks.workunit.client.0.smithi102.stderr:179175: 2024-08-19T14:46:14.869+0000 7f24c65c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:19.419 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-19T14:46:19.419 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:46:19.419 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T14:46:19.419 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:46:19.420 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:46:19.420 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:19.420 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:46:19.420 INFO:tasks.workunit.client.0.smithi102.stderr:179175: 0 2024-08-19T14:46:19.420 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:19.420 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:46:19.420 INFO:tasks.workunit.client.0.smithi102.stderr:179175: 2024-08-19T14:46:15.388+0000 7f24c65c4640 -1 Falling back to public interface 2024-08-19T14:46:19.420 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:19.420 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:19.420 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:19.420 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:46:19.420 INFO:tasks.workunit.client.0.smithi102.stderr:179175: 1 2024-08-19T14:46:19.420 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:19.421 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:46:19.421 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:19.421 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:19.421 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:19.421 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:46:19.421 INFO:tasks.workunit.client.0.smithi102.stderr:179175: 2 2024-08-19T14:46:19.421 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:19.421 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:46:19.421 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:19.421 INFO:tasks.workunit.client.0.smithi102.stderr:179175: 2024-08-19T14:46:18.150+0000 7f24c65c4640 -1 osd.2 60 log_to_monitors true 2024-08-19T14:46:19.421 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:19.421 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:19.421 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:46:19.421 INFO:tasks.workunit.client.0.smithi102.stderr:179175: 3 2024-08-19T14:46:19.422 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:19.422 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:46:20.023 INFO:tasks.workunit.client.0.smithi102.stderr:179175: osd.2 up in weight 1 up_from 67 up_thru 49 down_at 61 last_clean_interval [53,60) [v2:127.0.0.1:6810/2100614771,v1:127.0.0.1:6811/2100614771] [v2:127.0.0.1:6812/2100614771,v1:127.0.0.1:6813/2100614771] exists,up 49ecaa45-5329-4a2a-ac5b-16d876d456be 2024-08-19T14:46:20.023 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:46:20.023 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:46:20.023 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:46:20.023 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:46:20.023 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:46:20.024 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:46:20.024 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:46:20.024 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:46:20.024 INFO:tasks.workunit.client.0.smithi102.stderr:179173: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:46:20.024 INFO:tasks.workunit.client.0.smithi102.stderr:179173: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:46:20.024 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:46:20.024 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:46:20.024 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:46:20.024 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:46:20.024 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:46:20.024 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:46:20.024 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:46:20.024 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:46:20.025 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:46:20.025 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:46:20.025 INFO:tasks.workunit.client.0.smithi102.stderr:179173: 1 2024-08-19T14:46:20.025 INFO:tasks.workunit.client.0.smithi102.stderr:179173: 2 2024-08-19T14:46:20.025 INFO:tasks.workunit.client.0.smithi102.stderr:179173: 3' 2024-08-19T14:46:20.025 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:46:20.025 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:20.025 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:46:20.025 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836530 2024-08-19T14:46:20.025 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836530 2024-08-19T14:46:20.025 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530' 2024-08-19T14:46:20.025 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:20.025 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:46:20.025 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-08-19T14:46:20.026 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-08-19T14:46:20.026 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-283467841538' 2024-08-19T14:46:20.026 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph clean_interval [15,60) [v2:127.0.0.1:6818/724855685,v1:127.0.0.1:6819/724855685] [v2:127.0.0.1:6820/724855685,v1:127.0.0.1:6821/724855685] exists,up 23ffee92-771a-48be-9a5e-0328e959a8f1 2024-08-19T14:46:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:46:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:46:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:46:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:46:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:46:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:46:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:46:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:46:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:179175: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:46:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:179175: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:46:20.272 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:46:20.272 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:46:20.272 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:46:20.272 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:46:20.272 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:46:20.272 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:46:20.272 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:46:20.272 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:46:20.272 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:46:20.272 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:46:20.272 INFO:tasks.workunit.client.0.smithi102.stderr:179175: 1 2024-08-19T14:46:20.272 INFO:tasks.workunit.client.0.smithi102.stderr:179175: 2 2024-08-19T14:46:20.272 INFO:tasks.workunit.client.0.smithi102.stderr:179175: 3' 2024-08-19T14:46:20.272 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:46:20.273 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:20.273 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:46:20.273 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836531 2024-08-19T14:46:20.273 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836531 2024-08-19T14:46:20.273 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531' 2024-08-19T14:46:20.273 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:20.273 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:46:20.273 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841539 2024-08-19T14:46:20.273 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841539 2024-08-19T14:46:20.273 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-283467841539' 2024-08-19T14:46:20.273 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:22.298 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph telltell osd.2 flush_pg_stats 2024-08-19T14:46:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-08-19T14:46:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-08-19T14:46:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-283467841538 2-287762808834' 2024-08-19T14:46:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:46:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070469 2024-08-19T14:46:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070469 2024-08-19T14:46:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-283467841538 2-287762808834 3-253403070469' 2024-08-19T14:46:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836530 2024-08-19T14:46:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:22.300 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:46:22.300 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836530 2024-08-19T14:46:22.300 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:22.300 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836530 2024-08-19T14:46:22.300 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-08-19T14:46:22.300 INFO:tasks.workunit.client.0.smithi102.stderr:179173: waiting osd.0 seq 21474836530 2024-08-19T14:46:22.300 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:46:22.300 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836530 -lt 21474836530 2024-08-19T14:46:22.300 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:22.300 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-283467841538 2024-08-19T14:46:22.300 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:22.300 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:46:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-283467841538 2024-08-19T14:46:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-08-19T14:46:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 283467841538' 2024-08-19T14:46:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:179173: waiting osd.1 seq 283467841538 2024-08-19T14:46:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:46:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 283467841538 2024-08-19T14:46:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:46:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:46:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:46:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841539 -lt 283467841538 2024-08-19T14:46:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:22.461 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //hom osd.2 flush_pg_stats 2024-08-19T14:46:22.461 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808835 2024-08-19T14:46:22.461 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808835 2024-08-19T14:46:22.461 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-283467841539 2-287762808835' 2024-08-19T14:46:22.461 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:22.461 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:46:22.461 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070470 2024-08-19T14:46:22.462 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070470 2024-08-19T14:46:22.462 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-283467841539 2-287762808835 3-253403070470' 2024-08-19T14:46:22.462 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:22.462 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836531 2024-08-19T14:46:22.462 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:22.462 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:46:22.462 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836531 2024-08-19T14:46:22.462 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:22.462 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836531 2024-08-19T14:46:22.462 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2024-08-19T14:46:22.462 INFO:tasks.workunit.client.0.smithi102.stderr:179175: waiting osd.0 seq 21474836531 2024-08-19T14:46:22.462 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:46:22.462 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836530 -lt 21474836531 2024-08-19T14:46:22.463 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:46:22.463 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:46:22.463 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:46:22.463 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836531 -lt 21474836531 2024-08-19T14:46:22.463 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:22.463 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-283467841539 2024-08-19T14:46:22.463 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:22.463 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:46:22.463 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-283467841539 2024-08-19T14:46:22.463 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:22.463 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841539 2024-08-19T14:46:22.463 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 283467841539' 2024-08-19T14:46:22.463 INFO:tasks.workunit.client.0.smithi102.stderr:179175: waiting osd.1 seq 283467841539 2024-08-19T14:46:22.464 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:46:22.464 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841539 -lt 283467841539 2024-08-19T14:46:22.464 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:179175: e/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-287762808834 2024-08-19T14:46:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:46:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-287762808834 2024-08-19T14:46:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-08-19T14:46:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 287762808834' 2024-08-19T14:46:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:179173: waiting osd.2 seq 287762808834 2024-08-19T14:46:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:46:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808835 -lt 287762808834 2024-08-19T14:46:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070469 2024-08-19T14:46:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:46:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070469 2024-08-19T14:46:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070469 2024-08-19T14:46:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070469' 2024-08-19T14:46:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:179173: waiting osd.3 seq 253403070469 2024-08-19T14:46:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:46:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070470 -lt 253403070469 2024-08-19T14:46:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:46:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:46:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:46:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:46:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:46:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:46:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:46:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:46:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:46:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:46:23.512 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:46:23.512 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:46:23.512 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:46:23.719 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/q//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-287762808835 2024-08-19T14:46:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:46:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-287762808835 2024-08-19T14:46:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808835 2024-08-19T14:46:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 287762808835' 2024-08-19T14:46:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:179175: waiting osd.2 seq 287762808835 2024-08-19T14:46:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:46:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808835 -lt 287762808835 2024-08-19T14:46:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070470 2024-08-19T14:46:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:46:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070470 2024-08-19T14:46:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070470 2024-08-19T14:46:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070470' 2024-08-19T14:46:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:179175: waiting osd.3 seq 253403070470 2024-08-19T14:46:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:46:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070470 -lt 253403070470 2024-08-19T14:46:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:46:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:46:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:46:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:46:23.722 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:46:23.722 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:46:23.722 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:46:23.722 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:46:23.722 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:46:23.722 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:46:23.722 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:46:23.722 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:46:23.722 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:46:23.852 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.cliena/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:46:23.852 INFO:tasks.workunit.client.0.smithi102.stderr:179173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:46:23.852 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:46:23.852 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:46:23.852 INFO:tasks.workunit.client.0.smithi102.stderr:179173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:46:23.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:46:23.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:46:23.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 179172 2024-08-19T14:46:24.088 INFO:tasks.workunit.client.0.smithi102.stderr:t.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:46:24.089 INFO:tasks.workunit.client.0.smithi102.stderr:179175: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:46:24.089 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:46:24.089 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:46:24.089 INFO:tasks.workunit.client.0.smithi102.stderr:179175: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:46:24.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:46:24.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-08-19T14:46:24.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-08-19T14:46:24.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-08-19T14:46:24.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-08-19T14:46:24.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-08-19T14:46:24.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-08-19T14:46:24.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T14:46:24.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T14:46:24.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:46:24.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:46:24.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-08-19T14:46:24.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-08-19T14:46:24.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-08-19T14:46:24.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-08-19T14:46:24.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:46:24.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:46:24.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:46:24.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:46:24.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T14:45:57.477184+0000 2024-08-19T14:46:24.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-08-19T14:46:24.954 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-08-19T14:46:24.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-08-19T14:45:57.477184+0000 2024-08-19T14:46:24.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T14:46:24.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:45:57.477184+0000 2024-08-19T14:46:24.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:46:24.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:46:24.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:46:24.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:46:24.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:46:24.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:46:24.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:46:24.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:46:25.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:45:57.477184+0000 '>' 2024-08-19T14:45:57.477184+0000 2024-08-19T14:46:25.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:46:26.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:46:26.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:46:26.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:46:26.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:46:26.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:46:26.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:46:26.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:46:26.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:45:57.477184+0000 '>' 2024-08-19T14:45:57.477184+0000 2024-08-19T14:46:26.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:46:27.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:46:27.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:46:27.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:46:27.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:46:27.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:46:27.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:46:27.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:46:27.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:45:57.477184+0000 '>' 2024-08-19T14:45:57.477184+0000 2024-08-19T14:46:27.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:46:28.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:46:28.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:46:28.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:46:28.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:46:28.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:46:28.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:46:28.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:46:29.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:46:25.272827+0000 '>' 2024-08-19T14:45:57.477184+0000 2024-08-19T14:46:29.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:46:29.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-08-19T14:46:29.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:46:29.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:46:29.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:46:29.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 181997"' 2024-08-19T14:46:29.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 181997' 2024-08-19T14:46:29.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-08-19T14:46:29.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:46:29.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:46:29.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 181999"' 2024-08-19T14:46:29.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 181999' 2024-08-19T14:46:29.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:46:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-08-19T14:46:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 181997 181999' 2024-08-19T14:46:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-08-19T14:46:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:46:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 181997 2024-08-19T14:46:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/182000: /' 2024-08-19T14:46:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-08-19T14:46:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/182002: /' 2024-08-19T14:46:31.510 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:46:31.510 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:46:31.510 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-08-19T14:46:31.510 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:46:31.510 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-08-19T14:46:31.510 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:46:31.511 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:46:31.511 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-08-19T14:46:31.511 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:46:31.511 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-08-19T14:46:31.511 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:46:31.511 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:46:31.511 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:46:31.511 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:46:31.511 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:46:31.511 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:46:31.511 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-08-19T14:46:31.511 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:46:31.512 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:46:31.512 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-08-19T14:46:31.512 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:46:31.512 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-08-19T14:46:31.512 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-08-19T14:46:31.512 INFO:tasks.workunit.client.0.smithi102.stderr:182002: _ 2024-08-19T14:46:31.512 INFO:tasks.workunit.client.0.smithi102.stderr:182002: hinfo_key 2024-08-19T14:46:31.512 INFO:tasks.workunit.client.0.smithi102.stderr:182002: snapset 2024-08-19T14:46:31.512 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-08-19T14:46:31.512 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:46:31.512 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:46:31.512 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-19T14:46:31.512 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:46:31.512 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T14:46:31.513 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:46:31.513 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:46:31.513 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:46:31.513 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:46:31.513 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T14:46:31.513 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T14:46:31.513 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:46:31.513 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:46:31.514 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:46:31.514 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:46:31.514 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:46:31.514 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:46:31.514 INFO:tasks.workunit.client.0.smithi102.stderr:182002: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:46:31.514 INFO:tasks.workunit.client.0.smithi102.stderr:182002: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:46:31.514 INFO:tasks.workunit.client.0.smithi102.stderr:182002: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:46:31.514 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:46:31.514 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:46:31.514 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:46:31.514 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:46:31.514 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:46:31.515 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:46:31.515 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:46:31.515 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:46:31.515 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:46:31.515 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:46:31.515 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:46:31.515 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T14:46:31.515 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-19T14:46:31.515 INFO:tasks.workunit.client.0.smithi102.stderr:182002: start osd.2 2024-08-19T14:46:31.553 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:46:31.553 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:46:31.553 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T14:46:31.553 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:46:31.553 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:46:31.553 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:46:31.553 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:46:31.553 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T14:46:31.553 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:46:31.553 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T14:46:31.553 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:46:31.554 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:46:31.554 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:46:31.554 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:46:31.554 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:46:31.554 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:46:31.554 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:46:31.554 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:46:31.554 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:46:31.555 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T14:46:31.555 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:46:31.555 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:46:31.555 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-08-19T14:46:31.555 INFO:tasks.workunit.client.0.smithi102.stderr:182000: _ 2024-08-19T14:46:31.555 INFO:tasks.workunit.client.0.smithi102.stderr:182000: hinfo_key 2024-08-19T14:46:31.555 INFO:tasks.workunit.client.0.smithi102.stderr:182000: snapset 2024-08-19T14:46:31.555 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T14:46:31.555 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:46:31.555 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:46:31.555 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:46:31.555 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:46:31.556 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:46:31.556 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:46:31.556 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:46:31.556 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:46:31.556 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:46:31.556 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:46:31.557 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:46:31.557 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:46:31.557 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:46:31.557 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:46:31.557 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:46:31.557 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:46:31.558 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:46:31.558 INFO:tasks.workunit.client.0.smithi102.stderr:182000: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:46:31.558 INFO:tasks.workunit.client.0.smithi102.stderr:182000: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:46:31.558 INFO:tasks.workunit.client.0.smithi102.stderr:182000: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:46:31.558 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:46:31.558 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:46:31.558 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:46:31.558 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:46:31.558 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:46:31.558 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:46:31.558 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:46:31.558 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:46:31.559 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:46:31.559 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:46:31.559 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:46:31.559 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:46:31.559 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:46:31.559 INFO:tasks.workunit.client.0.smithi102.stderr:182000: start osd.1 2024-08-19T14:46:36.020 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:46:36.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-08-19T14:46:36.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-19T14:46:36.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:46:36.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:46:36.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:46:36.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: 2024-08-19T14:46:31.542+0000 7f40b35d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:36.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: 2024-08-19T14:46:31.545+0000 7f40b35d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:36.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: 2024-08-19T14:46:31.546+0000 7f40b35d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:36.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-19T14:46:36.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:46:36.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T14:46:36.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:46:36.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:46:36.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:36.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:46:36.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: 0 2024-08-19T14:46:36.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:36.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:46:36.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: 2024-08-19T14:46:32.061+0000 7f40b35d6640 -1 Falling back to public interface 2024-08-19T14:46:36.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:36.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:36.022 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:36.022 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:46:36.022 INFO:tasks.workunit.client.0.smithi102.stderr:182002: 1 2024-08-19T14:46:36.022 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:36.022 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:46:36.022 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:36.022 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:36.022 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:36.022 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:46:36.022 INFO:tasks.workunit.client.0.smithi102.stderr:182002: 2 2024-08-19T14:46:36.022 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:36.023 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:46:36.023 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:36.023 INFO:tasks.workunit.client.0.smithi102.stderr:182002: 2024-08-19T14:46:34.646+0000 7f40b35d6640 -1 osd.2 69 log_to_monitors true 2024-08-19T14:46:36.023 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:36.023 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:36.023 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:46:36.023 INFO:tasks.workunit.client.0.smithi102.stderr:182002: 3 2024-08-19T14:46:36.023 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:36.023 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:46:36.023 INFO:tasks.workunit.client.0.smithi102.stderr:182002: osd.2 up in weight 1 up_from 75 up_thru 49 down_aovery_ops 2024-08-19T14:46:36.023 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T14:46:36.023 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:46:36.023 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:46:36.023 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:46:36.024 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:46:36.024 INFO:tasks.workunit.client.0.smithi102.stderr:182000: 2024-08-19T14:46:31.585+0000 7f2c14c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:36.024 INFO:tasks.workunit.client.0.smithi102.stderr:182000: 2024-08-19T14:46:31.588+0000 7f2c14c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:36.024 INFO:tasks.workunit.client.0.smithi102.stderr:182000: 2024-08-19T14:46:31.589+0000 7f2c14c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:36.024 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:46:36.024 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:46:36.024 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:46:36.024 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:46:36.024 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:46:36.024 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:36.025 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:46:36.025 INFO:tasks.workunit.client.0.smithi102.stderr:182000: 0 2024-08-19T14:46:36.025 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:36.025 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:46:36.025 INFO:tasks.workunit.client.0.smithi102.stderr:182000: 2024-08-19T14:46:32.114+0000 7f2c14c15640 -1 Falling back to public interface 2024-08-19T14:46:36.025 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:36.025 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:36.025 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:36.025 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:46:36.025 INFO:tasks.workunit.client.0.smithi102.stderr:182000: 1 2024-08-19T14:46:36.025 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:36.026 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:46:36.026 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:36.026 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:36.026 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:36.026 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:46:36.026 INFO:tasks.workunit.client.0.smithi102.stderr:182000: 2 2024-08-19T14:46:36.026 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:36.026 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:46:36.026 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:36.026 INFO:tasks.workunit.client.0.smithi102.stderr:182000: 2024-08-19T14:46:34.786+0000 7f2c14c15640 -1 osd.1 69 log_to_monitors true 2024-08-19T14:46:36.026 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:36.026 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:36.027 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:46:36.027 INFO:tasks.workunit.client.0.smithi102.stderr:182000: 3 2024-08-19T14:46:36.027 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:36.027 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:46:36.905 INFO:tasks.workunit.client.0.smithi102.stderr:182000: osd.1 up in weight 1 up_from 75 up_thru 67 down_at 70 last_clean_interval [67,69) [v2:127.0.0.1:6810/2029252869,v1:127.0.0.1:6811/2029252869] [v2:127.0.0.1:6812/2029252869,v1:127.0.0.1:6813/2029252869] exists,up 23ffee92-771a-48be-9a5e-0328e959a8f1 2024-08-19T14:46:36.905 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:46:36.905 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:46:36.906 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:46:36.906 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:46:36.906 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:46:36.906 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:46:36.906 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:46:36.906 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:46:36.906 INFO:tasks.workunit.client.0.smithi102.stderr:182002: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:46:36.906 INFO:tasks.workunit.client.0.smithi102.stderr:182002: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:46:36.906 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:46:36.906 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:46:36.906 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:46:36.906 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:46:36.907 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:46:36.907 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:46:36.907 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:46:36.907 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:46:36.907 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:46:36.907 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:46:36.907 INFO:tasks.workunit.client.0.smithi102.stderr:182002: 1 2024-08-19T14:46:36.907 INFO:tasks.workunit.client.0.smithi102.stderr:182002: 2 2024-08-19T14:46:36.907 INFO:tasks.workunit.client.0.smithi102.stderr:182002: 3' 2024-08-19T14:46:36.907 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:46:36.907 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:36.907 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:46:36.908 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836535 2024-08-19T14:46:36.908 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836535 2024-08-19T14:46:36.908 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535' 2024-08-19T14:46:36.908 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:36.908 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:46:36.908 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547202 2024-08-19T14:46:36.908 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547202 2024-08-19T14:46:36.908 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-322122547202' 2024-08-19T14:46:36.908 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:36.908 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stat 70 last_clean_interval [66,69) [v2:127.0.0.1:6818/1080813966,v1:127.0.0.1:6819/1080813966] [v2:127.0.0.1:6820/1080813966,v1:127.0.0.1:6821/1080813966] exists,up 49ecaa45-5329-4a2a-ac5b-16d876d456be 2024-08-19T14:46:36.908 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:46:36.908 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:46:36.909 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:46:36.909 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:46:36.909 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:46:36.909 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:46:36.909 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:46:36.909 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:46:36.909 INFO:tasks.workunit.client.0.smithi102.stderr:182000: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:46:36.909 INFO:tasks.workunit.client.0.smithi102.stderr:182000: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:46:36.909 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:46:36.909 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:46:36.909 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:46:36.909 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:46:36.909 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:46:36.910 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:46:36.910 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:46:36.910 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:46:36.910 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:46:36.910 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:46:36.910 INFO:tasks.workunit.client.0.smithi102.stderr:182000: 1 2024-08-19T14:46:36.910 INFO:tasks.workunit.client.0.smithi102.stderr:182000: 2 2024-08-19T14:46:36.910 INFO:tasks.workunit.client.0.smithi102.stderr:182000: 3' 2024-08-19T14:46:36.910 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:46:36.910 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:36.910 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:46:36.911 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836536 2024-08-19T14:46:36.911 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836536 2024-08-19T14:46:36.911 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536' 2024-08-19T14:46:36.911 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:36.911 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:46:36.911 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547203 2024-08-19T14:46:36.911 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547203 2024-08-19T14:46:36.911 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-322122547203' 2024-08-19T14:46:36.911 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:39.011 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:46:39.011 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547203 2024-08-19T14:46:39.011 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547203 2024-08-19T14:46:39.011 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-322122547203 2-322122547203' 2024-08-19T14:46:39.011 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:39.011 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:46:39.012 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070474 2024-08-19T14:46:39.012 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070474 2024-08-19T14:46:39.012 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-322122547203 2-322122547203 3-253403070474' 2024-08-19T14:46:39.012 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:39.012 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836536 2024-08-19T14:46:39.012 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:39.013 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:46:39.013 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836536 2024-08-19T14:46:39.013 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:39.013 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836536 2024-08-19T14:46:39.013 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2024-08-19T14:46:39.013 INFO:tasks.workunit.client.0.smithi102.stderr:182000: waiting osd.0 seq 21474836536 2024-08-19T14:46:39.013 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:46:39.013 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836536 2024-08-19T14:46:39.013 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:46:39.013 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:46:39.013 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:46:39.014 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836537 -lt 21474836536 2024-08-19T14:46:39.014 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:39.014 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-322122547203 2024-08-19T14:46:39.014 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:39.014 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:46:39.014 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-322122547203 2024-08-19T14:46:39.014 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:39.014 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547203 2024-08-19T14:46:39.014 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 322122547203' 2024-08-19T14:46:39.014 INFO:tasks.workunit.client.0.smithi102.stderr:182000: waiting osd.1 seq 322122547203 2024-08-19T14:46:39.014 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:46:39.014 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547203 -lt 322122547203 2024-08-19T14:46:39.018 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in ts: ceph tell osd.2 flush_pg_stats 2024-08-19T14:46:39.019 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547202 2024-08-19T14:46:39.019 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547202 2024-08-19T14:46:39.019 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-322122547202 2-322122547202' 2024-08-19T14:46:39.019 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:39.019 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:46:39.019 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070475 2024-08-19T14:46:39.019 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070475 2024-08-19T14:46:39.019 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-322122547202 2-322122547202 3-253403070475' 2024-08-19T14:46:39.019 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:39.019 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836535 2024-08-19T14:46:39.019 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:39.019 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:46:39.019 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836535 2024-08-19T14:46:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836535 2024-08-19T14:46:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-08-19T14:46:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: waiting osd.0 seq 21474836535 2024-08-19T14:46:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:46:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836535 2024-08-19T14:46:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:46:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:46:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:46:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836537 -lt 21474836535 2024-08-19T14:46:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-322122547202 2024-08-19T14:46:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:46:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-322122547202 2024-08-19T14:46:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547202 2024-08-19T14:46:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 322122547202' 2024-08-19T14:46:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: waiting osd.1 seq 322122547202 2024-08-19T14:46:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:46:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547203 -lt 322122547202 2024-08-19T14:46:40.202 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:40.202 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547203 2024-08-19T14:46:40.202 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:40.202 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:46:40.203 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547203 2024-08-19T14:46:40.203 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:40.203 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547203 2024-08-19T14:46:40.203 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547203' 2024-08-19T14:46:40.203 INFO:tasks.workunit.client.0.smithi102.stderr:182000: waiting osd.2 seq 322122547203 2024-08-19T14:46:40.203 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:46:40.203 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547203 -lt 322122547203 2024-08-19T14:46:40.203 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:40.203 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070474 2024-08-19T14:46:40.203 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:40.203 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:46:40.203 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070474 2024-08-19T14:46:40.203 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:40.204 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070474 2024-08-19T14:46:40.204 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070474' 2024-08-19T14:46:40.204 INFO:tasks.workunit.client.0.smithi102.stderr:182000: waiting osd.3 seq 253403070474 2024-08-19T14:46:40.204 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:46:40.204 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070476 -lt 253403070474 2024-08-19T14:46:40.204 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:46:40.204 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:46:40.204 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:46:40.204 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:46:40.204 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:46:40.205 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:46:40.205 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:46:40.205 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:46:40.205 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:46:40.205 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:46:40.205 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:46:40.205 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:46:40.205 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:46:40.281 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephte$seqs 2024-08-19T14:46:40.282 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547202 2024-08-19T14:46:40.282 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:40.282 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:46:40.282 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547202 2024-08-19T14:46:40.282 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:40.282 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547202 2024-08-19T14:46:40.282 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547202' 2024-08-19T14:46:40.282 INFO:tasks.workunit.client.0.smithi102.stderr:182002: waiting osd.2 seq 322122547202 2024-08-19T14:46:40.282 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:46:40.282 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547203 -lt 322122547202 2024-08-19T14:46:40.283 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:40.283 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070475 2024-08-19T14:46:40.283 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:40.284 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:46:40.284 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070475 2024-08-19T14:46:40.285 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:40.285 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070475 2024-08-19T14:46:40.285 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070475' 2024-08-19T14:46:40.285 INFO:tasks.workunit.client.0.smithi102.stderr:182002: waiting osd.3 seq 253403070475 2024-08-19T14:46:40.285 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:46:40.285 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070476 -lt 253403070475 2024-08-19T14:46:40.285 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:46:40.285 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:46:40.285 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:46:40.286 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:46:40.286 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:46:40.286 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:46:40.286 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:46:40.286 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:46:40.286 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:46:40.286 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:46:40.286 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:46:40.291 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:46:40.291 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:46:40.545 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:46:40.545 INFO:tasks.workunit.client.0.smithi102.stderr:182000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:46:40.545 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:46:40.546 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:46:40.546 INFO:tasks.workunit.client.0.smithi102.stderr:182000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:46:40.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:46:40.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:46:40.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 181999 2024-08-19T14:46:40.626 INFO:tasks.workunit.client.0.smithi102.stderr:st/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:46:40.626 INFO:tasks.workunit.client.0.smithi102.stderr:182002: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:46:40.626 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:46:40.626 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:46:40.626 INFO:tasks.workunit.client.0.smithi102.stderr:182002: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:46:40.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:46:40.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-08-19T14:46:40.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-08-19T14:46:40.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-08-19T14:46:40.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-08-19T14:46:40.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-08-19T14:46:40.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T14:46:40.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T14:46:40.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-08-19T14:46:40.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-08-19T14:46:40.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-08-19T14:46:40.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-08-19T14:46:40.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-08-19T14:46:40.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-08-19T14:46:40.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:46:40.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:46:40.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:46:40.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 184676"' 2024-08-19T14:46:40.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 184676' 2024-08-19T14:46:40.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:46:40.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:46:40.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:46:40.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 184678"' 2024-08-19T14:46:40.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:46:40.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 184678' 2024-08-19T14:46:40.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-08-19T14:46:40.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 184676 184678' 2024-08-19T14:46:40.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-08-19T14:46:40.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:46:40.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 184676 2024-08-19T14:46:40.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/184679: /' 2024-08-19T14:46:40.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:46:40.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/184681: /' 2024-08-19T14:46:43.206 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:46:43.206 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:46:43.207 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T14:46:43.207 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:46:43.207 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:46:43.207 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:46:43.207 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:46:43.207 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T14:46:43.207 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:46:43.207 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T14:46:43.207 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:46:43.207 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:46:43.207 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:46:43.207 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:46:43.208 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:46:43.208 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:46:43.208 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T14:46:43.208 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:46:43.208 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:46:43.208 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T14:46:43.208 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:46:43.208 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:46:43.208 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-08-19T14:46:43.208 INFO:tasks.workunit.client.0.smithi102.stderr:184681: remove 1#2:eb822e21:::SOMETHING:head# 2024-08-19T14:46:43.208 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T14:46:43.208 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:46:43.209 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:46:43.209 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:46:43.209 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:46:43.209 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:46:43.209 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:46:43.209 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:46:43.210 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:46:43.210 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:46:43.210 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:46:43.210 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:46:43.210 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:46:43.210 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:46:43.210 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:46:43.210 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:46:43.210 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:46:43.210 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:46:43.211 INFO:tasks.workunit.client.0.smithi102.stderr:184681: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:46:43.211 INFO:tasks.workunit.client.0.smithi102.stderr:184681: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:46:43.211 INFO:tasks.workunit.client.0.smithi102.stderr:184681: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:46:43.211 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:46:43.211 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:46:43.211 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:46:43.211 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:46:43.211 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:46:43.211 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:46:43.211 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:46:43.211 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:46:43.211 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:46:43.211 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:46:43.212 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:46:43.212 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:46:43.212 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:46:43.212 INFO:tasks.workunit.client.0.smithi102.stderr:184681: start osd.1 2024-08-19T14:46:43.212 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:46:43.700 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:46:43.700 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:46:43.700 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-08-19T14:46:43.700 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:46:43.701 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:46:43.701 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:46:43.701 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:46:43.701 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-08-19T14:46:43.701 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:46:43.701 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-08-19T14:46:43.701 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:46:43.701 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:46:43.701 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:46:43.701 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:46:43.701 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:46:43.701 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:46:43.701 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:46:43.702 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:46:43.702 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:46:43.702 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-08-19T14:46:43.702 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:46:43.702 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:46:43.702 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-08-19T14:46:43.702 INFO:tasks.workunit.client.0.smithi102.stderr:184679: remove 0#2:eb822e21:::SOMETHING:head# 2024-08-19T14:46:43.702 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-08-19T14:46:43.703 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:46:43.703 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:46:43.703 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-19T14:46:43.703 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:46:43.703 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:46:43.703 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:46:43.703 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:46:43.703 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:46:43.703 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:46:43.704 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T14:46:43.704 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T14:46:43.704 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:46:43.704 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:46:43.704 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:46:43.704 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:46:43.704 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:46:43.704 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:46:43.704 INFO:tasks.workunit.client.0.smithi102.stderr:184679: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:46:43.704 INFO:tasks.workunit.client.0.smithi102.stderr:184679: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:46:43.704 INFO:tasks.workunit.client.0.smithi102.stderr:184679: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:46:43.704 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:46:43.705 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:46:43.705 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:46:43.705 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:46:43.705 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:46:43.705 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:46:43.705 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:46:43.705 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:46:43.705 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:46:43.705 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:46:43.705 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:46:43.705 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T14:46:43.705 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-19T14:46:43.705 INFO:tasks.workunit.client.0.smithi102.stderr:184679: start osd.3 2024-08-19T14:46:43.706 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:46:47.655 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T14:46:47.656 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:46:47.656 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:46:47.656 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:46:47.656 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:46:47.656 INFO:tasks.workunit.client.0.smithi102.stderr:184681: 2024-08-19T14:46:43.238+0000 7f586ef88640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:47.656 INFO:tasks.workunit.client.0.smithi102.stderr:184681: 2024-08-19T14:46:43.241+0000 7f586ef88640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:47.656 INFO:tasks.workunit.client.0.smithi102.stderr:184681: 2024-08-19T14:46:43.242+0000 7f586ef88640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:47.656 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:46:47.656 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:46:47.656 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:46:47.656 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:46:47.657 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:46:47.657 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:47.657 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:46:47.657 INFO:tasks.workunit.client.0.smithi102.stderr:184681: 0 2024-08-19T14:46:47.657 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:47.657 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:46:47.657 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:47.657 INFO:tasks.workunit.client.0.smithi102.stderr:184681: 2024-08-19T14:46:43.767+0000 7f586ef88640 -1 Falling back to public interface 2024-08-19T14:46:47.657 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:47.657 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:47.657 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:46:47.657 INFO:tasks.workunit.client.0.smithi102.stderr:184681: 1 2024-08-19T14:46:47.658 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:47.658 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:46:47.658 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:47.658 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:47.658 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:47.658 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:46:47.658 INFO:tasks.workunit.client.0.smithi102.stderr:184681: 2 2024-08-19T14:46:47.658 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:47.658 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:46:47.658 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:47.658 INFO:tasks.workunit.client.0.smithi102.stderr:184681: 2024-08-19T14:46:46.324+0000 7f586ef88640 -1 osd.1 77 log_to_monitors true 2024-08-19T14:46:47.658 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:47.658 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:47.659 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:46:47.659 INFO:tasks.workunit.client.0.smithi102.stderr:184681: 3 2024-08-19T14:46:47.659 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:47.659 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:46:48.340 INFO:tasks.workunit.client.0.smithi102.stderr:184681: osd.1 up in weight 1 up_from 82 up_thru 82 down_at 78 last_184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-08-19T14:46:48.341 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-19T14:46:48.341 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:46:48.341 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:46:48.341 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:46:48.341 INFO:tasks.workunit.client.0.smithi102.stderr:184679: 2024-08-19T14:46:43.728+0000 7fb63d43d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:48.341 INFO:tasks.workunit.client.0.smithi102.stderr:184679: 2024-08-19T14:46:43.730+0000 7fb63d43d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:48.341 INFO:tasks.workunit.client.0.smithi102.stderr:184679: 2024-08-19T14:46:43.731+0000 7fb63d43d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:46:48.341 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-19T14:46:48.341 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:46:48.341 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T14:46:48.341 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:46:48.341 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:46:48.341 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:48.342 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:46:48.342 INFO:tasks.workunit.client.0.smithi102.stderr:184679: 0 2024-08-19T14:46:48.342 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:48.342 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:46:48.342 INFO:tasks.workunit.client.0.smithi102.stderr:184679: 2024-08-19T14:46:44.251+0000 7fb63d43d640 -1 Falling back to public interface 2024-08-19T14:46:48.342 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:48.342 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:48.342 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:48.342 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:46:48.342 INFO:tasks.workunit.client.0.smithi102.stderr:184679: 1 2024-08-19T14:46:48.342 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:48.342 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:46:48.342 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:48.343 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:48.343 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:48.343 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:46:48.343 INFO:tasks.workunit.client.0.smithi102.stderr:184679: 2 2024-08-19T14:46:48.343 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:48.343 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:46:48.343 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:46:48.343 INFO:tasks.workunit.client.0.smithi102.stderr:184679: 2024-08-19T14:46:47.073+0000 7fb63d43d640 -1 osd.3 77 log_to_monitors true 2024-08-19T14:46:48.343 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:46:48.343 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:46:48.343 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:46:48.343 INFO:tasks.workunit.client.0.smithi102.stderr:184679: 3 2024-08-19T14:46:48.343 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:46:48.343 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:46:48.473 INFO:tasks.workunit.client.0.smithi102.stderr:184679: osd.3 up in weight 1 up_from 85 up_thru 75 down_at 78 last_clean_interval [75,77) [v2:127.0.0.1:6818/286613263,v1:127.0.0.1:6819/286613263] [v2:127.0.0.1:6820/286613263,v1:127.0.0.1:6821/286613263] exists,up 49ecaa45-5329-4a2a-ac5b-16d876d456be 2024-08-19T14:46:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:46:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:46:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:46:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:46:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:46:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:46:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:46:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:46:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:184681: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:46:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:184681: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:46:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:46:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:46:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:46:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:46:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:46:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:46:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:46:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:46:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:46:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:46:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:184681: 1 2024-08-19T14:46:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:184681: 2 2024-08-19T14:46:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:184681: 3' 2024-08-19T14:46:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:46:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:46:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836540 2024-08-19T14:46:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836540 2024-08-19T14:46:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540' 2024-08-19T14:46:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:46:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-08-19T14:46:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-08-19T14:46:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-352187318274' 2024-08-19T14:46:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:49.021 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tellclean_interval [59,77) [v2:127.0.0.1:6826/366629461,v1:127.0.0.1:6827/366629461] [v2:127.0.0.1:6828/366629461,v1:127.0.0.1:6829/366629461] exists,up f2573c78-0f5b-410e-9c19-12dc1c78b131 2024-08-19T14:46:49.021 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:46:49.021 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:46:49.021 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:46:49.021 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:46:49.021 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:46:49.022 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:46:49.022 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:46:49.022 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:46:49.022 INFO:tasks.workunit.client.0.smithi102.stderr:184679: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:46:49.022 INFO:tasks.workunit.client.0.smithi102.stderr:184679: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:46:49.022 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:46:49.022 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:46:49.022 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:46:49.022 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:46:49.022 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:46:49.022 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:46:49.022 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:46:49.022 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:46:49.023 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:46:49.023 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:46:49.023 INFO:tasks.workunit.client.0.smithi102.stderr:184679: 1 2024-08-19T14:46:49.023 INFO:tasks.workunit.client.0.smithi102.stderr:184679: 2 2024-08-19T14:46:49.023 INFO:tasks.workunit.client.0.smithi102.stderr:184679: 3' 2024-08-19T14:46:49.023 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:46:49.023 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:49.023 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:46:49.023 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836541 2024-08-19T14:46:49.023 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836541 2024-08-19T14:46:49.023 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541' 2024-08-19T14:46:49.023 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:49.023 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:46:49.024 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318275 2024-08-19T14:46:49.024 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318275 2024-08-19T14:46:49.024 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-352187318275' 2024-08-19T14:46:49.024 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:50.686 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:46:50.687 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547206 2024-08-19T14:46:50.687 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547206 2024-08-19T14:46:50.687 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-352187318274 2-322122547206' 2024-08-19T14:46:50.687 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:50.687 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:46:50.687 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220162 2024-08-19T14:46:50.687 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220162 2024-08-19T14:46:50.687 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-352187318274 2-322122547206 3-365072220162' 2024-08-19T14:46:50.687 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:50.687 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836540 2024-08-19T14:46:50.687 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:50.688 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:46:50.688 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836540 2024-08-19T14:46:50.688 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:50.688 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836540 2024-08-19T14:46:50.688 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-08-19T14:46:50.688 INFO:tasks.workunit.client.0.smithi102.stderr:184681: waiting osd.0 seq 21474836540 2024-08-19T14:46:50.688 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:46:50.688 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836539 -lt 21474836540 2024-08-19T14:46:50.688 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:46:50.688 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:46:50.688 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:46:50.688 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836540 2024-08-19T14:46:50.688 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:50.689 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-352187318274 2024-08-19T14:46:50.689 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:50.689 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:46:50.689 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-352187318274 2024-08-19T14:46:50.689 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:50.689 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-08-19T14:46:50.689 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 352187318274' 2024-08-19T14:46:50.689 INFO:tasks.workunit.client.0.smithi102.stderr:184681: waiting osd.1 seq 352187318274 2024-08-19T14:46:50.689 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:46:50.689 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318275 -lt 352187318274 2024-08-19T14:46:50.689 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:51.112 INFO:tasks.workunit.client.0.smithi102.stderr:184681: osd.2 flush_pg_stats 2024-08-19T14:46:51.112 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547207 2024-08-19T14:46:51.112 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547207 2024-08-19T14:46:51.112 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-352187318275 2-322122547207' 2024-08-19T14:46:51.112 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:46:51.112 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:46:51.112 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220163 2024-08-19T14:46:51.113 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220163 2024-08-19T14:46:51.113 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-352187318275 2-322122547207 3-365072220163' 2024-08-19T14:46:51.113 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:51.113 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836541 2024-08-19T14:46:51.113 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:51.113 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:46:51.113 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836541 2024-08-19T14:46:51.113 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:51.113 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836541 2024-08-19T14:46:51.113 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2024-08-19T14:46:51.113 INFO:tasks.workunit.client.0.smithi102.stderr:184679: waiting osd.0 seq 21474836541 2024-08-19T14:46:51.113 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:46:51.113 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836539 -lt 21474836541 2024-08-19T14:46:51.113 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:46:51.114 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:46:51.114 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:46:51.114 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836541 2024-08-19T14:46:51.114 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:51.114 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-352187318275 2024-08-19T14:46:51.114 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:51.114 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:46:51.114 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-352187318275 2024-08-19T14:46:51.114 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:51.114 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318275 2024-08-19T14:46:51.114 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 352187318275' 2024-08-19T14:46:51.114 INFO:tasks.workunit.client.0.smithi102.stderr:184679: waiting osd.1 seq 352187318275 2024-08-19T14:46:51.114 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:46:51.115 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318275 -lt 352187318275 2024-08-19T14:46:51.115 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:51.877 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547206 2024-08-19T14:46:51.877 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:51.877 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:46:51.877 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547206 2024-08-19T14:46:51.877 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:51.877 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547206 2024-08-19T14:46:51.878 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547206' 2024-08-19T14:46:51.878 INFO:tasks.workunit.client.0.smithi102.stderr:184681: waiting osd.2 seq 322122547206 2024-08-19T14:46:51.878 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:46:51.878 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547207 -lt 322122547206 2024-08-19T14:46:51.878 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:51.878 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-365072220162 2024-08-19T14:46:51.878 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:51.878 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:46:51.878 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-365072220162 2024-08-19T14:46:51.878 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:51.878 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220162 2024-08-19T14:46:51.878 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 365072220162' 2024-08-19T14:46:51.879 INFO:tasks.workunit.client.0.smithi102.stderr:184681: waiting osd.3 seq 365072220162 2024-08-19T14:46:51.879 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:46:51.879 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220163 -lt 365072220162 2024-08-19T14:46:51.879 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:46:51.879 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:46:51.879 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:46:51.879 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:46:51.879 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:46:51.879 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:46:51.879 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:46:51.879 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:46:51.879 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:46:51.879 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:46:51.880 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:46:51.880 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:46:51.880 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:46:52.318 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.clien//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547207 2024-08-19T14:46:52.318 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:52.318 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:46:52.318 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547207 2024-08-19T14:46:52.319 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:52.319 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547207 2024-08-19T14:46:52.319 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547207' 2024-08-19T14:46:52.319 INFO:tasks.workunit.client.0.smithi102.stderr:184679: waiting osd.2 seq 322122547207 2024-08-19T14:46:52.319 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:46:52.319 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547207 -lt 322122547207 2024-08-19T14:46:52.319 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:46:52.319 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-365072220163 2024-08-19T14:46:52.319 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:46:52.319 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:46:52.319 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-365072220163 2024-08-19T14:46:52.319 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:46:52.319 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220163 2024-08-19T14:46:52.320 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 365072220163' 2024-08-19T14:46:52.320 INFO:tasks.workunit.client.0.smithi102.stderr:184679: waiting osd.3 seq 365072220163 2024-08-19T14:46:52.320 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:46:52.320 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220163 -lt 365072220163 2024-08-19T14:46:52.320 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:46:52.320 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:46:52.320 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:46:52.320 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:46:52.320 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:46:52.320 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:46:52.320 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:46:52.320 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:46:52.320 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:46:52.321 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:46:52.321 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:46:52.321 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:46:52.321 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:46:53.309 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:46:53.309 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:46:53.309 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-08-19T14:46:53.309 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-08-19T14:46:53.309 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:46:53.309 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-08-19T14:46:53.309 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:46:53.310 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:46:53.310 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:46:53.310 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:46:53.310 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:46:53.310 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:46:53.310 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:46:53.310 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:46:53.310 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:46:53.310 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:46:53.310 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:46:53.310 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:46:53.310 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:46:53.311 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:46:53.311 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-08-19T14:46:53.311 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-08-19T14:46:53.311 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T14:46:53.311 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T14:46:53.311 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T14:46:53.311 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T14:46:53.311 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T14:46:53.311 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T14:46:53.311 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T14:46:53.311 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T14:46:53.311 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T14:46:53.737 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clont.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:46:53.737 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:46:53.738 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-08-19T14:46:53.738 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-08-19T14:46:53.738 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:46:53.738 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-08-19T14:46:53.738 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:46:53.738 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:46:53.738 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:46:53.738 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:46:53.738 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:46:53.738 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:46:53.738 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:46:53.738 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:46:53.739 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:46:53.739 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:46:53.739 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:46:53.739 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:46:53.739 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:46:53.739 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:46:53.739 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-08-19T14:46:53.739 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-08-19T14:46:53.739 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T14:46:53.739 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T14:46:53.739 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T14:46:53.739 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T14:46:53.739 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T14:46:53.740 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T14:46:53.740 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T14:46:53.740 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T14:46:53.740 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T14:46:54.580 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-19T14:46:54.581 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:46:54.581 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-19T14:46:54.581 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:46:54.581 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:46:54.581 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:46:54.581 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:46:54.581 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:46:54.581 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:46:54.581 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:46:54.581 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:46:54.581 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:46:54.582 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:46:54.582 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:46:54.582 INFO:tasks.workunit.client.0.smithi102.stderr:184679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:46:54.582 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:46:54.582 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:46:54.582 INFO:tasks.workunit.client.0.smithi102.stderr:184679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:46:54.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:46:54.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:46:54.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 184678 2024-08-19T14:46:54.899 INFO:tasks.workunit.client.0.smithi102.stderr:e.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-19T14:46:54.899 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:46:54.899 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-19T14:46:54.899 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:46:54.899 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:46:54.899 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:46:54.899 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:46:54.900 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:46:54.900 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:46:54.900 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:46:54.900 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:46:54.900 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:46:54.900 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:46:54.900 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:46:54.900 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:46:54.900 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-08-19T14:46:54.901 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-08-19T14:46:54.901 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T14:46:54.901 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T14:46:54.901 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T14:46:54.901 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T14:46:54.901 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T14:46:54.901 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T14:46:54.901 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T14:46:54.901 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T14:46:54.901 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T14:46:54.901 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-08-19T14:46:54.902 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:46:54.902 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-19T14:46:54.902 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:46:54.902 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:46:55.493 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:46:55.493 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:46:55.494 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:46:55.494 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:46:55.494 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:46:55.494 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:46:55.494 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:46:55.494 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:46:55.494 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:46:55.494 INFO:tasks.workunit.client.0.smithi102.stderr:184681: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:46:55.494 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:46:55.494 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:46:55.494 INFO:tasks.workunit.client.0.smithi102.stderr:184681: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:46:55.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:46:55.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-08-19T14:46:55.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-08-19T14:46:55.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-08-19T14:46:55.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-08-19T14:46:55.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-08-19T14:46:55.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-08-19T14:46:55.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T14:46:55.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T14:46:55.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:46:55.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:46:55.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-08-19T14:46:55.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-08-19T14:46:55.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-08-19T14:46:55.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-08-19T14:46:55.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:46:55.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:46:55.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:46:55.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:46:56.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T14:46:25.272827+0000 2024-08-19T14:46:56.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-08-19T14:46:56.306 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-08-19T14:46:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-08-19T14:46:25.272827+0000 2024-08-19T14:46:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T14:46:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:46:25.272827+0000 2024-08-19T14:46:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:46:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:46:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:46:56.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:46:56.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:46:56.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:46:56.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:46:56.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:46:56.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:46:25.272827+0000 '>' 2024-08-19T14:46:25.272827+0000 2024-08-19T14:46:56.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:46:57.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:46:57.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:46:57.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:46:57.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:46:57.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:46:57.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:46:57.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:46:57.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:46:25.272827+0000 '>' 2024-08-19T14:46:25.272827+0000 2024-08-19T14:46:57.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:46:58.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:46:58.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:46:58.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:46:58.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:46:58.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:46:58.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:46:58.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:46:59.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:46:57.035440+0000 '>' 2024-08-19T14:46:25.272827+0000 2024-08-19T14:46:59.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:46:59.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-08-19T14:46:59.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:46:59.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:46:59.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:46:59.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 187795"' 2024-08-19T14:46:59.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 187795' 2024-08-19T14:46:59.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:46:59.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:46:59.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:46:59.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 187797"' 2024-08-19T14:46:59.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 187797' 2024-08-19T14:46:59.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:46:59.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-08-19T14:46:59.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 187795 187797' 2024-08-19T14:46:59.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-08-19T14:46:59.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:46:59.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 187795 2024-08-19T14:46:59.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/187798: /' 2024-08-19T14:46:59.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:46:59.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/187800: /' 2024-08-19T14:47:01.520 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:47:01.520 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:47:01.520 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-08-19T14:47:01.520 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:47:01.520 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:47:01.520 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:47:01.520 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:47:01.520 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-08-19T14:47:01.521 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:47:01.521 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-08-19T14:47:01.521 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:47:01.521 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:47:01.521 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:47:01.521 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:47:01.521 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:47:01.521 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:47:01.521 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:47:01.521 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:47:01.521 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:47:01.521 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-08-19T14:47:01.521 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:47:01.522 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:47:01.522 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-08-19T14:47:01.522 INFO:tasks.workunit.client.0.smithi102.stderr:187798: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-08-19T14:47:01.522 INFO:tasks.workunit.client.0.smithi102.stderr:187798: _ 2024-08-19T14:47:01.522 INFO:tasks.workunit.client.0.smithi102.stderr:187798: hinfo_key 2024-08-19T14:47:01.522 INFO:tasks.workunit.client.0.smithi102.stderr:187798: snapset 2024-08-19T14:47:01.522 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-08-19T14:47:01.522 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:47:01.522 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:47:01.522 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-19T14:47:01.522 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:47:01.522 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:47:01.523 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:47:01.523 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:47:01.523 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:47:01.523 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:47:01.523 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T14:47:01.523 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T14:47:01.523 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:47:01.523 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:47:01.524 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:47:01.524 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:47:01.524 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:47:01.524 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:47:01.524 INFO:tasks.workunit.client.0.smithi102.stderr:187798: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:47:01.524 INFO:tasks.workunit.client.0.smithi102.stderr:187798: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:47:01.524 INFO:tasks.workunit.client.0.smithi102.stderr:187798: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:47:01.524 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:47:01.524 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:47:01.524 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:47:01.524 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:47:01.524 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:47:01.524 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:47:01.525 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:47:01.525 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:47:01.525 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:47:01.525 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:47:01.525 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:47:01.525 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T14:47:01.525 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-19T14:47:01.525 INFO:tasks.workunit.client.0.smithi102.stderr:187798: start osd.3 2024-08-19T14:47:01.896 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:47:01.896 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:47:01.896 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T14:47:01.896 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:47:01.896 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:47:01.896 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:47:01.896 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:47:01.896 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T14:47:01.896 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:47:01.897 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T14:47:01.897 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:47:01.897 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:47:01.897 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:47:01.897 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:47:01.897 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:47:01.897 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:47:01.897 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T14:47:01.897 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:47:01.897 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:47:01.897 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T14:47:01.897 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:47:01.897 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:47:01.898 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-08-19T14:47:01.898 INFO:tasks.workunit.client.0.smithi102.stderr:187800: _ 2024-08-19T14:47:01.898 INFO:tasks.workunit.client.0.smithi102.stderr:187800: hinfo_key 2024-08-19T14:47:01.898 INFO:tasks.workunit.client.0.smithi102.stderr:187800: snapset 2024-08-19T14:47:01.898 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T14:47:01.898 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:47:01.898 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:47:01.898 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T14:47:01.898 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:47:01.898 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:47:01.898 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:47:01.898 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:47:01.899 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:47:01.899 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:47:01.899 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:47:01.899 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:47:01.899 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:47:01.899 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:47:01.899 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:47:01.899 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:47:01.899 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:47:01.900 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:47:01.900 INFO:tasks.workunit.client.0.smithi102.stderr:187800: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:47:01.900 INFO:tasks.workunit.client.0.smithi102.stderr:187800: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:47:01.900 INFO:tasks.workunit.client.0.smithi102.stderr:187800: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:47:01.900 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:47:01.900 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:47:01.900 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:47:01.900 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:47:01.900 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:47:01.900 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:47:01.900 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:47:01.900 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:47:01.900 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:47:01.901 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:47:01.901 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:47:01.901 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:47:01.901 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T14:47:01.901 INFO:tasks.workunit.client.0.smithi102.stderr:187800: start osd.1 2024-08-19T14:47:05.639 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:47:05.639 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-08-19T14:47:05.639 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-19T14:47:05.639 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:47:05.639 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:47:05.639 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:47:05.640 INFO:tasks.workunit.client.0.smithi102.stderr:187798: 2024-08-19T14:47:01.552+0000 7fc1ccc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:47:05.640 INFO:tasks.workunit.client.0.smithi102.stderr:187798: 2024-08-19T14:47:01.555+0000 7fc1ccc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:47:05.640 INFO:tasks.workunit.client.0.smithi102.stderr:187798: 2024-08-19T14:47:01.556+0000 7fc1ccc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:47:05.640 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-19T14:47:05.640 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:47:05.640 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T14:47:05.640 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:47:05.640 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:47:05.640 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:05.640 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:47:05.640 INFO:tasks.workunit.client.0.smithi102.stderr:187798: 0 2024-08-19T14:47:05.640 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:05.640 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:47:05.640 INFO:tasks.workunit.client.0.smithi102.stderr:187798: 2024-08-19T14:47:02.070+0000 7fc1ccc0d640 -1 Falling back to public interface 2024-08-19T14:47:05.641 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:05.641 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:05.641 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:05.641 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:47:05.641 INFO:tasks.workunit.client.0.smithi102.stderr:187798: 1 2024-08-19T14:47:05.641 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:05.641 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:47:05.641 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:05.641 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:05.641 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:05.641 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:47:05.641 INFO:tasks.workunit.client.0.smithi102.stderr:187798: 2 2024-08-19T14:47:05.641 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:05.642 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:47:05.642 INFO:tasks.workunit.client.0.smithi102.stderr:187798: 2024-08-19T14:47:04.500+0000 7fc1ccc0d640 -1 osd.3 87 log_to_monitors true 2024-08-19T14:47:05.642 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:05.642 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:05.642 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:05.642 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:47:05.642 INFO:tasks.workunit.client.0.smithi102.stderr:187798: 3 2024-08-19T14:47:05.642 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:06.478 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-08-19T14:47:06.478 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T14:47:06.478 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T14:47:06.478 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:47:06.478 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:47:06.478 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:47:06.478 INFO:tasks.workunit.client.0.smithi102.stderr:187800: 2024-08-19T14:47:01.924+0000 7ff60dcdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:47:06.478 INFO:tasks.workunit.client.0.smithi102.stderr:187800: 2024-08-19T14:47:01.939+0000 7ff60dcdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:47:06.478 INFO:tasks.workunit.client.0.smithi102.stderr:187800: 2024-08-19T14:47:01.941+0000 7ff60dcdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:47:06.479 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T14:47:06.479 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:47:06.479 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:47:06.479 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:47:06.479 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:47:06.479 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:06.479 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:47:06.479 INFO:tasks.workunit.client.0.smithi102.stderr:187800: 0 2024-08-19T14:47:06.479 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:06.479 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:47:06.479 INFO:tasks.workunit.client.0.smithi102.stderr:187800: 2024-08-19T14:47:02.455+0000 7ff60dcdd640 -1 Falling back to public interface 2024-08-19T14:47:06.479 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:06.479 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:06.480 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:06.480 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:47:06.480 INFO:tasks.workunit.client.0.smithi102.stderr:187800: 1 2024-08-19T14:47:06.480 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:06.480 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:47:06.480 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:06.480 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:06.480 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:06.480 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:47:06.480 INFO:tasks.workunit.client.0.smithi102.stderr:187800: 2 2024-08-19T14:47:06.480 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:06.480 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:47:06.480 INFO:tasks.workunit.client.0.smithi102.stderr:187800: 2024-08-19T14:47:05.113+0000 7ff60dcdd640 -1 osd.1 87 log_to_monitors true 2024-08-19T14:47:06.481 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:06.481 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:06.481 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:06.481 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:47:06.481 INFO:tasks.workunit.client.0.smithi102.stderr:187800: 3 2024-08-19T14:47:06.481 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:06.481 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:47:06.867 INFO:tasks.workunit.client.0.smithi102.stderr:187800: osd.1 up in weight 1 up_from 94 up_thru 94 down_as.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:47:06.867 INFO:tasks.workunit.client.0.smithi102.stderr:187798: osd.3 up in weight 1 up_from 92 up_thru 92 down_at 88 last_clean_interval [85,87) [v2:127.0.0.1:6818/2531245950,v1:127.0.0.1:6819/2531245950] [v2:127.0.0.1:6820/2531245950,v1:127.0.0.1:6821/2531245950] exists,up f2573c78-0f5b-410e-9c19-12dc1c78b131 2024-08-19T14:47:06.867 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:47:06.867 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:47:06.867 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:47:06.867 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:47:06.867 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:47:06.867 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:47:06.867 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:47:06.867 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:47:06.867 INFO:tasks.workunit.client.0.smithi102.stderr:187798: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:47:06.867 INFO:tasks.workunit.client.0.smithi102.stderr:187798: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:47:06.868 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:47:06.868 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:47:06.868 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:47:06.868 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:47:06.868 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:47:06.868 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:47:06.868 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:47:06.868 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:47:06.868 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:47:06.868 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:47:06.868 INFO:tasks.workunit.client.0.smithi102.stderr:187798: 1 2024-08-19T14:47:06.868 INFO:tasks.workunit.client.0.smithi102.stderr:187798: 2 2024-08-19T14:47:06.868 INFO:tasks.workunit.client.0.smithi102.stderr:187798: 3' 2024-08-19T14:47:06.869 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:47:06.869 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:47:06.869 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:47:06.869 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836545 2024-08-19T14:47:06.869 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836545 2024-08-19T14:47:06.869 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545' 2024-08-19T14:47:06.869 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:47:06.869 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:47:06.869 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925826 2024-08-19T14:47:06.869 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925826 2024-08-19T14:47:06.869 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545 1-403726925826' 2024-08-19T14:47:07.107 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd it 88 last_clean_interval [82,87) [v2:127.0.0.1:6826/4197089229,v1:127.0.0.1:6827/4197089229] [v2:127.0.0.1:6828/4197089229,v1:127.0.0.1:6829/4197089229] exists,up 49ecaa45-5329-4a2a-ac5b-16d876d456be 2024-08-19T14:47:07.108 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:47:07.108 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:47:07.108 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:47:07.108 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:47:07.108 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:47:07.108 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:47:07.108 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:47:07.108 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:47:07.108 INFO:tasks.workunit.client.0.smithi102.stderr:187800: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:47:07.108 INFO:tasks.workunit.client.0.smithi102.stderr:187800: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:47:07.108 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:47:07.108 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:47:07.109 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:47:07.109 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:47:07.109 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:47:07.109 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:47:07.109 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:47:07.109 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:47:07.109 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:47:07.109 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:47:07.109 INFO:tasks.workunit.client.0.smithi102.stderr:187800: 1 2024-08-19T14:47:07.109 INFO:tasks.workunit.client.0.smithi102.stderr:187800: 2 2024-08-19T14:47:07.109 INFO:tasks.workunit.client.0.smithi102.stderr:187800: 3' 2024-08-19T14:47:07.109 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:47:07.109 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:47:07.109 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:47:07.110 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836546 2024-08-19T14:47:07.110 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836546 2024-08-19T14:47:07.110 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546' 2024-08-19T14:47:07.110 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:47:07.110 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:47:07.110 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925827 2024-08-19T14:47:07.110 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925827 2024-08-19T14:47:07.110 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-403726925827' 2024-08-19T14:47:07.110 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:47:08.963 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stan $ids 2024-08-19T14:47:08.963 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:47:08.963 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547212 2024-08-19T14:47:08.963 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547212 2024-08-19T14:47:08.963 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545 1-403726925826 2-322122547212' 2024-08-19T14:47:08.963 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:47:08.963 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:47:08.964 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991234 2024-08-19T14:47:08.964 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991234 2024-08-19T14:47:08.964 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545 1-403726925826 2-322122547212 3-395136991234' 2024-08-19T14:47:08.964 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:47:08.964 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836545 2024-08-19T14:47:08.964 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:47:08.964 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:47:08.964 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836545 2024-08-19T14:47:08.964 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:47:08.964 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836545 2024-08-19T14:47:08.964 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836545' 2024-08-19T14:47:08.964 INFO:tasks.workunit.client.0.smithi102.stderr:187798: waiting osd.0 seq 21474836545 2024-08-19T14:47:08.965 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:47:08.965 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836544 -lt 21474836545 2024-08-19T14:47:08.965 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:47:08.965 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:47:08.965 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:47:08.965 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836547 -lt 21474836545 2024-08-19T14:47:08.965 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:47:08.965 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-403726925826 2024-08-19T14:47:08.965 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:47:08.965 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:47:08.965 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-403726925826 2024-08-19T14:47:08.966 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:47:08.966 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925826 2024-08-19T14:47:08.966 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 403726925826' 2024-08-19T14:47:08.966 INFO:tasks.workunit.client.0.smithi102.stderr:187798: waiting osd.1 seq 403726925826 2024-08-19T14:47:08.966 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:47:08.966 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925826 2024-08-19T14:47:09.151 INFO:tasks.workunit.client.0.smithi102.stderr:187798ts: ceph tell osd.2 flush_pg_stats 2024-08-19T14:47:09.151 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547213 2024-08-19T14:47:09.151 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547213 2024-08-19T14:47:09.152 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-403726925827 2-322122547213' 2024-08-19T14:47:09.152 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:47:09.152 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:47:09.152 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991235 2024-08-19T14:47:09.152 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991235 2024-08-19T14:47:09.152 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-403726925827 2-322122547213 3-395136991235' 2024-08-19T14:47:09.152 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:47:09.153 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836546 2024-08-19T14:47:09.153 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:47:09.153 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:47:09.153 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836546 2024-08-19T14:47:09.153 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:47:09.154 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836546 2024-08-19T14:47:09.154 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836546' 2024-08-19T14:47:09.154 INFO:tasks.workunit.client.0.smithi102.stderr:187800: waiting osd.0 seq 21474836546 2024-08-19T14:47:09.154 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:47:09.154 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836544 -lt 21474836546 2024-08-19T14:47:09.155 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:47:09.155 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:47:09.155 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:47:09.155 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836547 -lt 21474836546 2024-08-19T14:47:09.155 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:47:09.155 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-403726925827 2024-08-19T14:47:09.156 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:47:09.156 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:47:09.156 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-403726925827 2024-08-19T14:47:09.156 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:47:09.156 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925827 2024-08-19T14:47:09.156 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 403726925827' 2024-08-19T14:47:09.157 INFO:tasks.workunit.client.0.smithi102.stderr:187800: waiting osd.1 seq 403726925827 2024-08-19T14:47:09.157 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:47:09.157 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925827 2024-08-19T14:47:10.183 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in : /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:47:10.183 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547212 2024-08-19T14:47:10.183 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:47:10.183 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:47:10.184 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547212 2024-08-19T14:47:10.184 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:47:10.184 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547212 2024-08-19T14:47:10.184 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547212' 2024-08-19T14:47:10.184 INFO:tasks.workunit.client.0.smithi102.stderr:187798: waiting osd.2 seq 322122547212 2024-08-19T14:47:10.184 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:47:10.184 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547213 -lt 322122547212 2024-08-19T14:47:10.184 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:47:10.184 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-395136991234 2024-08-19T14:47:10.184 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:47:10.184 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:47:10.184 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-395136991234 2024-08-19T14:47:10.184 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:47:10.184 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991234 2024-08-19T14:47:10.185 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 395136991234' 2024-08-19T14:47:10.185 INFO:tasks.workunit.client.0.smithi102.stderr:187798: waiting osd.3 seq 395136991234 2024-08-19T14:47:10.185 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:47:10.185 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991235 -lt 395136991234 2024-08-19T14:47:10.185 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:47:10.185 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:47:10.185 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:47:10.185 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:47:10.185 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:47:10.185 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:47:10.185 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:47:10.185 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:47:10.185 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:47:10.185 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:47:10.186 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:47:10.186 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:47:10.423 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.c$seqs 2024-08-19T14:47:10.423 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547213 2024-08-19T14:47:10.424 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:47:10.424 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:47:10.424 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547213 2024-08-19T14:47:10.424 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:47:10.424 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547213 2024-08-19T14:47:10.424 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547213' 2024-08-19T14:47:10.424 INFO:tasks.workunit.client.0.smithi102.stderr:187800: waiting osd.2 seq 322122547213 2024-08-19T14:47:10.424 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:47:10.424 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547213 -lt 322122547213 2024-08-19T14:47:10.424 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:47:10.424 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-395136991235 2024-08-19T14:47:10.424 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:47:10.424 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:47:10.425 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-395136991235 2024-08-19T14:47:10.425 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:47:10.425 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991235 2024-08-19T14:47:10.425 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 395136991235' 2024-08-19T14:47:10.425 INFO:tasks.workunit.client.0.smithi102.stderr:187800: waiting osd.3 seq 395136991235 2024-08-19T14:47:10.425 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:47:10.425 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991235 -lt 395136991235 2024-08-19T14:47:10.425 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:47:10.425 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:47:10.425 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:47:10.425 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:47:10.425 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:47:10.425 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:47:10.426 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:47:10.426 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:47:10.426 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:47:10.426 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:47:10.426 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:47:10.426 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:47:10.426 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:47:10.556 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtelient.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:47:10.556 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:47:10.556 INFO:tasks.workunit.client.0.smithi102.stderr:187798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:47:10.556 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:47:10.556 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:47:10.556 INFO:tasks.workunit.client.0.smithi102.stderr:187798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:47:10.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:47:10.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:47:10.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 187797 2024-08-19T14:47:10.762 INFO:tasks.workunit.client.0.smithi102.stderr:st/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:47:10.762 INFO:tasks.workunit.client.0.smithi102.stderr:187800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:47:10.762 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:47:10.762 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:47:10.762 INFO:tasks.workunit.client.0.smithi102.stderr:187800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:47:10.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:47:10.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-08-19T14:47:10.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-08-19T14:47:10.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-08-19T14:47:10.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-08-19T14:47:10.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-08-19T14:47:10.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T14:47:10.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T14:47:10.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T14:47:10.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:47:10.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:47:10.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:47:10.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:47:10.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:47:10.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:47:10.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:47:10.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:47:10.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:47:10.920 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:47:10.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:47:10.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:47:10.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:47:10.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:47:10.922 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:47:10.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:47:10.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:47:10.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:47:10.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:47:10.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:47:10.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:47:10.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:47:10.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:47:10.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:47:10.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:47:10.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:47:10.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:47:10.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:47:10.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T14:47:10.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T14:47:10.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T14:47:10.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T14:47:10.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:47:10.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:47:10.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:47:10.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:47:10.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:47:10.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:47:10.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:47:10.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:47:10.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:47:10.959 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:47:10.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:47:10.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:47:10.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:47:10.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:47:10.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:47:10.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:47:10.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:47:10.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:47:10.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:47:10.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:47:10.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:47:10.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:47:10.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:47:10.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:47:10.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:47:10.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:47:10.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:47:10.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:47:10.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T14:47:10.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T14:47:10.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:47:10.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:47:10.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T14:47:10.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T14:47:10.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T14:47:10.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T14:47:10.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T14:47:10.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2024-08-19T14:47:10.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:842: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2024-08-19T14:47:10.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-08-19T14:47:10.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=false 2024-08-19T14:47:10.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-08-19T14:47:10.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-08-19T14:47:10.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T14:47:10.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T14:47:10.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T14:47:10.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T14:47:10.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T14:47:10.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-08-19T14:47:11.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T14:47:11.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:47:11.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:47:11.347 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:47:11.347 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:47:11.347 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:47:11.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:47:11.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T14:47:11.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T14:47:11.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T14:47:11.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:47:11.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:47:11.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T14:47:11.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T14:47:11.541 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:47:11.541 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:47:11.541 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:47:11.541 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:47:11.541 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:47:11.542 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:47:11.542 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:47:11.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:47:11.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T14:47:11.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T14:47:11.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:47:11.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:47:11.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T14:47:11.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T14:47:11.620 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:47:11.621 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:47:11.621 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:47:11.621 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:47:11.621 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:47:11.621 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:47:11.621 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:47:11.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:47:11.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T14:47:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-08-19T14:47:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T14:47:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T14:47:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T14:47:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T14:47:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T14:47:11.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T14:47:12.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T14:47:12.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:47:12.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:47:12.508 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:47:12.508 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:47:12.508 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:47:12.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:47:12.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:47:12.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:47:12.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-08-19T14:47:12.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T14:47:12.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-08-19T14:47:12.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:47:12.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:47:12.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T14:47:12.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:47:12.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:47:12.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:47:12.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:47:12.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:47:12.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:47:12.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:47:12.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:47:12.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:47:12.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:47:12.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:47:12.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:47:12.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:47:12.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:47:12.542 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:47:12.542 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:47:12.543 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:47:12.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:47:12.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:47:12.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:47:12.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:47:12.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:47:12.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:47:12.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:47:12.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:47:12.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:47:12.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:47:12.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:47:12.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:47:12.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:47:12.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:47:12.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:47:12.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bb58a600-8ebe-469d-b139-c46800d2832f 2024-08-19T14:47:12.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bb58a600-8ebe-469d-b139-c46800d2832f' 2024-08-19T14:47:12.544 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 bb58a600-8ebe-469d-b139-c46800d2832f 2024-08-19T14:47:12.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:47:12.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDwWsNmli9bIRAAArXr8jSBa3lbDI6EqLZZgA== 2024-08-19T14:47:12.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDwWsNmli9bIRAAArXr8jSBa3lbDI6EqLZZgA=="}' 2024-08-19T14:47:12.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bb58a600-8ebe-469d-b139-c46800d2832f -i td/osd-scrub-repair/0/new.json 2024-08-19T14:47:12.764 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:47:12.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T14:47:12.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDwWsNmli9bIRAAArXr8jSBa3lbDI6EqLZZgA== --osd-uuid bb58a600-8ebe-469d-b139-c46800d2832f 2024-08-19T14:47:12.803 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:12.803+0000 7f47f2e16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:47:12.815 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:12.814+0000 7f47f2e16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:47:12.816 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:12.815+0000 7f47f2e16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:47:12.816 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:12.815+0000 7f47f2e16640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:47:12.816 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:12.815+0000 7f47f2e16640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:47:12.816 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:12.815+0000 7f47f2e16640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:47:12.890 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:12.889+0000 7f47f2e16640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T14:47:18.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T14:47:18.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:47:18.206 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T14:47:18.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T14:47:18.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:47:18.549 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T14:47:18.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T14:47:18.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:47:18.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:47:18.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:47:18.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:47:18.576 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:18.575+0000 7f9ff161b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:47:18.604 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:18.603+0000 7f9ff161b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:47:18.605 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:18.604+0000 7f9ff161b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:47:18.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T14:47:18.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:47:18.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:47:18.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:47:18.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:47:18.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:18.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:47:18.861 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:47:18.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:18.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:47:19.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:19.150 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:19.150+0000 7f9ff161b640 -1 Falling back to public interface 2024-08-19T14:47:20.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:20.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:20.126 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:47:20.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:47:20.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:20.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:47:20.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:21.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:21.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:21.397 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:47:21.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:47:21.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:21.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:47:21.413 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:21.412+0000 7f9ff161b640 -1 osd.0 0 log_to_monitors true 2024-08-19T14:47:21.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:22.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:22.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:22.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:47:22.690 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:47:22.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:22.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:47:22.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:23.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:23.963 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:47:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:47:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:47:24.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:25.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:25.306 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:47:25.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:25.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:47:25.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:25.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:47:25.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:26.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:26.585 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:47:26.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:26.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:47:26.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:26.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:47:26.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:27.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:27.861 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:47:27.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:27.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:47:27.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:27.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:47:28.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:29.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:29.148 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:47:29.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:29.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:47:29.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:29.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:47:29.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:30.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:30.428 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:47:30.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:30.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:47:30.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:30.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:47:30.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:31.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:31.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:31.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:47:31.702 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:47:31.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:31.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:47:31.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:32.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:32.983 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:47:32.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:32.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:47:32.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:32.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:47:33.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:34.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:34.331 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:47:34.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:34.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:47:34.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:34.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:47:34.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:35.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:35.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:35.744 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:47:35.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:47:35.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:35.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:47:36.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:37.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:37.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:37.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:47:37.286 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:47:37.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:37.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:47:37.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:38.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:38.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:38.905 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:47:38.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:47:38.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:38.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:47:39.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:40.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:40.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:40.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:47:40.691 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:47:40.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:40.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:47:41.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:42.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:42.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:42.452 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:47:42.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:47:42.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:42.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:47:42.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:43.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:43.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:43.796 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:47:43.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:47:43.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:43.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:47:44.063 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1462438138,v1:127.0.0.1:6803/1462438138] [v2:127.0.0.1:6804/1462438138,v1:127.0.0.1:6805/1462438138] exists,up bb58a600-8ebe-469d-b139-c46800d2832f 2024-08-19T14:47:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:47:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:47:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:47:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T14:47:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-08-19T14:47:44.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:47:44.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:47:44.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T14:47:44.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:47:44.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:47:44.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:47:44.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:47:44.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:47:44.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:47:44.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:47:44.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:47:44.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:47:44.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:47:44.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:47:44.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:47:44.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:47:44.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:47:44.072 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:47:44.072 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:47:44.072 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:47:44.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:47:44.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:47:44.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:47:44.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:47:44.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:47:44.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:47:44.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:47:44.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:47:44.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:47:44.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:47:44.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:47:44.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:47:44.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:47:44.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:47:44.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:47:44.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=396c1641-3fe7-4a6a-bb6d-743fdee8b5cf 2024-08-19T14:47:44.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 396c1641-3fe7-4a6a-bb6d-743fdee8b5cf' 2024-08-19T14:47:44.074 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 396c1641-3fe7-4a6a-bb6d-743fdee8b5cf 2024-08-19T14:47:44.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:47:44.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAQW8NmMO8MBRAAudVUSWG5PnUWuWQyfuA6lg== 2024-08-19T14:47:44.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAQW8NmMO8MBRAAudVUSWG5PnUWuWQyfuA6lg=="}' 2024-08-19T14:47:44.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 396c1641-3fe7-4a6a-bb6d-743fdee8b5cf -i td/osd-scrub-repair/1/new.json 2024-08-19T14:47:44.438 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:47:44.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T14:47:44.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAQW8NmMO8MBRAAudVUSWG5PnUWuWQyfuA6lg== --osd-uuid 396c1641-3fe7-4a6a-bb6d-743fdee8b5cf 2024-08-19T14:47:44.479 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:44.478+0000 7efdf0f44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:47:44.515 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:44.514+0000 7efdf0f44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:47:44.516 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:44.515+0000 7efdf0f44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:47:44.516 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:44.515+0000 7efdf0f44640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:47:44.516 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:44.515+0000 7efdf0f44640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:47:44.516 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:44.515+0000 7efdf0f44640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:47:44.591 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:44.590+0000 7efdf0f44640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T14:47:50.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T14:47:50.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:47:50.477 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T14:47:50.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T14:47:50.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:47:50.822 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T14:47:50.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T14:47:50.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:47:50.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:47:50.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:47:50.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:47:50.849 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:50.848+0000 7f9beac2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:47:50.850 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:50.850+0000 7f9beac2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:47:50.852 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:50.851+0000 7f9beac2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:47:51.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T14:47:51.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:47:51.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:47:51.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:47:51.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:47:51.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:51.088 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:47:51.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:47:51.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:51.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:47:51.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:51.377 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:51.377+0000 7f9beac2e640 -1 Falling back to public interface 2024-08-19T14:47:52.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:52.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:52.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:47:52.355 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:47:52.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:52.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:47:52.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:53.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:53.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:53.661 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:47:53.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:47:53.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:53.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:47:53.707 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:47:53.706+0000 7f9beac2e640 -1 osd.1 0 log_to_monitors true 2024-08-19T14:47:53.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:54.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:54.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:54.938 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:47:54.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:47:54.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:54.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:47:55.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:56.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:56.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:56.254 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:47:56.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:47:56.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:56.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:47:56.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:57.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:57.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:57.531 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:47:57.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:47:57.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:57.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:47:57.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:47:58.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:47:58.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:47:58.807 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:47:58.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:47:58.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:47:58.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:47:59.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:00.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:00.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:00.207 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:48:00.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:48:00.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:00.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:48:00.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:01.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:01.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:01.556 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:48:01.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:48:01.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:01.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:48:01.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:02.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:02.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:02.832 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:48:02.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:48:02.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:02.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:48:03.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:04.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:04.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:04.193 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:48:04.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:48:04.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:04.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:48:04.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:05.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:05.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:05.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:48:05.526 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:48:05.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:05.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:48:05.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:06.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:06.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:06.882 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:48:06.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:48:06.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:06.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:48:07.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:08.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:08.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:08.500 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:48:08.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:48:08.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:08.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:48:08.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:09.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:09.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:09.836 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:48:09.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:48:09.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:09.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:48:10.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:11.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:11.505 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:48:11.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:11.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:48:11.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:11.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:48:12.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:13.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:13.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:13.132 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:48:13.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:48:13.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:13.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:48:13.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:14.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:14.760 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:48:14.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:14.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:48:14.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:14.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:48:15.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:16.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:16.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:16.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:48:16.166 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:48:16.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:16.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:48:16.436 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2294503883,v1:127.0.0.1:6811/2294503883] [v2:127.0.0.1:6812/2294503883,v1:127.0.0.1:6813/2294503883] exists,up 396c1641-3fe7-4a6a-bb6d-743fdee8b5cf 2024-08-19T14:48:16.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:48:16.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:48:16.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:48:16.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T14:48:16.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-08-19T14:48:16.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:48:16.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:48:16.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T14:48:16.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:48:16.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T14:48:16.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:48:16.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:48:16.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:48:16.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:48:16.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T14:48:16.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T14:48:16.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:48:16.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:48:16.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:48:16.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:48:16.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:48:16.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:48:16.442 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:48:16.442 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:48:16.442 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:48:16.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:48:16.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:48:16.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:48:16.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:48:16.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:48:16.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:48:16.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:48:16.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:48:16.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:48:16.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:48:16.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:48:16.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:48:16.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:48:16.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T14:48:16.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:48:16.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e153c950-03a1-45b7-8ed4-81e4f0a7e3c6 2024-08-19T14:48:16.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e153c950-03a1-45b7-8ed4-81e4f0a7e3c6' 2024-08-19T14:48:16.444 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 e153c950-03a1-45b7-8ed4-81e4f0a7e3c6 2024-08-19T14:48:16.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:48:16.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAwW8Nm+aBMGxAACK2++yqs4xSsTOEL2a1MaQ== 2024-08-19T14:48:16.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAwW8Nm+aBMGxAACK2++yqs4xSsTOEL2a1MaQ=="}' 2024-08-19T14:48:16.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e153c950-03a1-45b7-8ed4-81e4f0a7e3c6 -i td/osd-scrub-repair/2/new.json 2024-08-19T14:48:16.781 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:48:16.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-08-19T14:48:16.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAwW8Nm+aBMGxAACK2++yqs4xSsTOEL2a1MaQ== --osd-uuid e153c950-03a1-45b7-8ed4-81e4f0a7e3c6 2024-08-19T14:48:16.822 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:16.821+0000 7f6aa67cb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:48:16.858 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:16.857+0000 7f6aa67cb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:48:16.858 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:16.858+0000 7f6aa67cb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:48:16.859 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:16.858+0000 7f6aa67cb640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:48:16.859 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:16.858+0000 7f6aa67cb640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:48:16.859 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:16.858+0000 7f6aa67cb640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:48:16.958 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:16.958+0000 7f6aa67cb640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-08-19T14:48:22.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-08-19T14:48:22.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:48:22.702 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T14:48:22.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T14:48:22.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:48:23.045 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T14:48:23.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T14:48:23.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:48:23.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:48:23.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:48:23.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:48:23.072 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:23.070+0000 7f10e39a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:48:23.074 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:23.072+0000 7f10e39a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:48:23.075 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:23.073+0000 7f10e39a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:48:23.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T14:48:23.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:48:23.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T14:48:23.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:48:23.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:48:23.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:23.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:48:23.313 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:48:23.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:23.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:48:23.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:23.591 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:23.590+0000 7f10e39a3640 -1 Falling back to public interface 2024-08-19T14:48:24.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:24.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:24.580 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:48:24.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:48:24.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:24.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:48:24.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:25.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:25.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:25.857 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:48:25.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:48:25.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:25.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:48:25.931 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:25.930+0000 7f10e39a3640 -1 osd.2 0 log_to_monitors true 2024-08-19T14:48:26.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:27.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:27.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:27.129 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:48:27.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:48:27.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:27.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:48:27.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:28.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:28.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:28.407 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:48:28.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:48:28.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:28.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:48:28.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:29.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:29.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:29.719 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:48:29.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:48:29.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:29.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:48:29.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:30.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:30.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:30.994 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:48:30.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:48:30.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:30.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:48:31.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:32.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:32.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:32.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:48:32.281 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:48:32.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:32.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:48:32.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:33.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:33.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:33.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:48:33.669 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:48:33.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:33.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:48:33.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:34.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:34.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:34.949 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:48:34.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:48:34.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:34.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:48:35.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:36.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:36.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:36.243 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:48:36.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:48:36.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:36.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:48:36.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:37.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:37.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:37.522 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:48:37.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:48:37.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:37.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:48:38.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:39.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:39.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:39.120 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:48:39.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:48:39.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:39.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:48:39.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:40.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:40.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:40.995 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:48:40.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:48:40.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:40.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:48:41.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:42.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:42.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:42.613 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:48:42.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:48:42.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:42.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:48:43.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:44.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:44.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:44.500 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:48:44.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:48:44.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:44.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:48:44.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:45.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:45.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:45.946 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:48:45.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:48:45.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:45.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:48:46.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:47.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:47.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:47.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:48:47.638 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:48:47.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:47.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:48:47.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:48.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:48.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:48.951 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:48:48.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:48:48.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:48.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:48:49.251 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3468842584,v1:127.0.0.1:6819/3468842584] [v2:127.0.0.1:6820/3468842584,v1:127.0.0.1:6821/3468842584] exists,up e153c950-03a1-45b7-8ed4-81e4f0a7e3c6 2024-08-19T14:48:49.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:48:49.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:48:49.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:48:49.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T14:48:49.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-08-19T14:48:49.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:48:49.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:48:49.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-08-19T14:48:49.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:48:49.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:48:49.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:48:49.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:48:49.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:48:49.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:48:49.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T14:48:49.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T14:48:49.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:48:49.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:48:49.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:48:49.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:48:49.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:48:49.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:48:49.256 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:48:49.257 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:48:49.257 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:48:49.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:48:49.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:48:49.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:48:49.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:48:49.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:48:49.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:48:49.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:48:49.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:48:49.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:48:49.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:48:49.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:48:49.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:48:49.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:48:49.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T14:48:49.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:48:49.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0a924bc7-1b05-4420-8781-adc75bdf85fc 2024-08-19T14:48:49.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 0a924bc7-1b05-4420-8781-adc75bdf85fc' 2024-08-19T14:48:49.258 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 0a924bc7-1b05-4420-8781-adc75bdf85fc 2024-08-19T14:48:49.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:48:49.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBRW8Nm4j0iEBAADA1JfUfysJDVoyDti1oL3A== 2024-08-19T14:48:49.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBRW8Nm4j0iEBAADA1JfUfysJDVoyDti1oL3A=="}' 2024-08-19T14:48:49.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0a924bc7-1b05-4420-8781-adc75bdf85fc -i td/osd-scrub-repair/3/new.json 2024-08-19T14:48:49.609 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:48:49.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-08-19T14:48:49.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBRW8Nm4j0iEBAADA1JfUfysJDVoyDti1oL3A== --osd-uuid 0a924bc7-1b05-4420-8781-adc75bdf85fc 2024-08-19T14:48:49.650 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:49.649+0000 7f49a5443640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:48:49.677 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:49.677+0000 7f49a5443640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:48:49.678 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:49.678+0000 7f49a5443640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:48:49.679 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:49.678+0000 7f49a5443640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-08-19T14:48:49.679 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:49.678+0000 7f49a5443640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-08-19T14:48:49.679 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:49.678+0000 7f49a5443640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-08-19T14:48:49.752 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:49.752+0000 7f49a5443640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-08-19T14:48:55.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-08-19T14:48:55.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:48:55.551 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-08-19T14:48:55.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-08-19T14:48:55.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:48:55.899 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-08-19T14:48:55.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-08-19T14:48:55.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:48:55.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:48:55.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:48:55.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:48:55.925 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:55.925+0000 7faaaeab5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:48:55.927 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:55.927+0000 7faaaeab5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:48:55.928 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:55.928+0000 7faaaeab5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:48:56.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-08-19T14:48:56.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:48:56.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T14:48:56.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:48:56.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:48:56.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:56.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:48:56.208 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:48:56.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:56.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:48:56.452 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:56.452+0000 7faaaeab5640 -1 Falling back to public interface 2024-08-19T14:48:56.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:57.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:57.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:57.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:48:57.473 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:48:57.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:57.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:48:57.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:48:58.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:48:58.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:48:58.751 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:48:58.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:48:58.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:48:58.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:48:58.969 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:48:58.968+0000 7faaaeab5640 -1 osd.3 0 log_to_monitors true 2024-08-19T14:48:59.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:00.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:00.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:00.023 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:49:00.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:49:00.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:00.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:49:00.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:01.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:01.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:01.296 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:49:01.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:49:01.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:01.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:49:01.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:02.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:02.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:02.578 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:49:02.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:49:02.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:02.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:49:03.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:04.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:04.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:04.027 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:49:04.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:49:04.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:04.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:49:04.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:05.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:05.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:05.310 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:49:05.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:49:05.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:05.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:49:05.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:06.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:06.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:06.590 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:49:06.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:49:06.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:06.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:49:06.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:07.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:07.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:07.992 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:49:07.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:49:07.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:07.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:49:08.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:09.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:09.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:09.267 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:49:09.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:49:09.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:09.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:49:09.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:10.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:10.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:10.583 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:49:10.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:49:10.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:10.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:49:11.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:12.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:12.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:12.228 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:49:12.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:49:12.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:12.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:49:12.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:13.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:13.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:13.943 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:49:13.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:49:13.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:13.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:49:14.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:15.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:15.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:15.466 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:49:15.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:49:15.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:15.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:49:16.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:17.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:17.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:17.032 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:49:17.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:49:17.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:17.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:49:17.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:18.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:18.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:18.847 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:49:18.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:49:18.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:18.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:49:19.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:20.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:20.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:20.595 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:49:20.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:49:20.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:20.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:49:21.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:22.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:22.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:22.049 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:49:22.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:49:22.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:22.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:49:22.325 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/282390753,v1:127.0.0.1:6827/282390753] [v2:127.0.0.1:6828/282390753,v1:127.0.0.1:6829/282390753] exists,up 0a924bc7-1b05-4420-8781-adc75bdf85fc 2024-08-19T14:49:22.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:49:22.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:49:22.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:49:22.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T14:49:22.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-08-19T14:49:22.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:49:22.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:49:22.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-08-19T14:49:22.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:49:22.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-08-19T14:49:22.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:49:22.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:49:22.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:49:22.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:49:22.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-08-19T14:49:22.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-08-19T14:49:22.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:49:22.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:49:22.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:49:22.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:49:22.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:49:22.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:49:22.331 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:49:22.331 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:49:22.331 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:49:22.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:49:22.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:49:22.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:49:22.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:49:22.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:49:22.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:49:22.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:49:22.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:49:22.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:49:22.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:49:22.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:49:22.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:49:22.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:49:22.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-08-19T14:49:22.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:49:22.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8b90df17-7dea-427e-b377-0de3d193ba0d 2024-08-19T14:49:22.334 INFO:tasks.workunit.client.0.smithi102.stdout:add osd4 8b90df17-7dea-427e-b377-0de3d193ba0d 2024-08-19T14:49:22.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 8b90df17-7dea-427e-b377-0de3d193ba0d' 2024-08-19T14:49:22.334 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:49:22.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQByW8NmILqrFBAAWQtlDElN97DTwZJ0gaRC6Q== 2024-08-19T14:49:22.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQByW8NmILqrFBAAWQtlDElN97DTwZJ0gaRC6Q=="}' 2024-08-19T14:49:22.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8b90df17-7dea-427e-b377-0de3d193ba0d -i td/osd-scrub-repair/4/new.json 2024-08-19T14:49:22.744 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:49:22.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-08-19T14:49:22.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQByW8NmILqrFBAAWQtlDElN97DTwZJ0gaRC6Q== --osd-uuid 8b90df17-7dea-427e-b377-0de3d193ba0d 2024-08-19T14:49:22.784 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:49:22.783+0000 7fe5b6553640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:49:22.837 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:49:22.837+0000 7fe5b6553640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:49:22.838 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:49:22.838+0000 7fe5b6553640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:49:22.839 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:49:22.838+0000 7fe5b6553640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-08-19T14:49:22.839 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:49:22.838+0000 7fe5b6553640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-08-19T14:49:22.839 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:49:22.838+0000 7fe5b6553640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-08-19T14:49:22.912 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:49:22.911+0000 7fe5b6553640 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-08-19T14:49:28.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-08-19T14:49:28.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:49:28.599 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd4 key to auth repository 2024-08-19T14:49:28.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-08-19T14:49:28.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:49:28.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-08-19T14:49:28.944 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.4 2024-08-19T14:49:28.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:49:28.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:49:28.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:49:28.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:49:28.970 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:49:28.970+0000 7f7cd953e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:49:28.972 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:49:28.972+0000 7f7cd953e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:49:28.973 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:49:28.973+0000 7f7cd953e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:49:29.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-08-19T14:49:29.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:49:29.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-08-19T14:49:29.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:49:29.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:49:29.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:29.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:49:29.211 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:49:29.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:29.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:49:29.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:29.507 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:49:29.507+0000 7f7cd953e640 -1 Falling back to public interface 2024-08-19T14:49:30.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:30.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:30.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:49:30.476 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:49:30.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:30.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:49:30.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:31.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:31.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:31.758 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:49:31.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:49:31.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:31.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:49:31.830 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:49:31.829+0000 7f7cd953e640 -1 osd.4 0 log_to_monitors true 2024-08-19T14:49:32.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:33.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:33.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:33.043 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:49:33.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:49:33.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:33.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:49:33.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:34.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:34.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:34.438 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:49:34.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:49:34.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:34.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:49:34.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:35.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:35.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:35.904 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:49:35.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:49:35.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:35.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:49:36.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:37.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:37.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:37.180 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:49:37.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:49:37.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:37.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:49:37.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:38.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:38.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:38.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:49:38.457 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:49:38.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:38.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:49:38.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:39.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:39.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:39.869 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:49:39.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:49:39.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:39.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:49:40.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:41.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:41.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:41.144 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:49:41.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:49:41.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:41.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:49:41.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:42.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:42.454 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:49:42.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:42.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:49:42.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:42.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:49:42.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:43.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:43.885 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:49:43.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:43.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:49:43.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:43.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:49:44.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:45.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:45.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:45.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:49:45.174 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:49:45.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:45.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:49:46.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:47.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:47.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:47.053 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:49:47.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:49:47.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:47.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:49:47.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:48.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:48.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:48.665 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:49:48.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:49:48.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:48.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:49:49.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:50.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:50.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:50.259 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:49:50.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:49:50.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:50.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:49:50.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:51.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:51.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:51.779 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:49:51.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:49:51.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:51.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:49:52.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:53.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:53.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:53.480 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:49:53.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:49:53.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:53.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:49:54.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:49:55.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:49:55.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:49:55.185 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:49:55.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:49:55.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:49:55.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:49:55.494 INFO:tasks.workunit.client.0.smithi102.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2646963421,v1:127.0.0.1:6835/2646963421] [v2:127.0.0.1:6836/2646963421,v1:127.0.0.1:6837/2646963421] exists,up 8b90df17-7dea-427e-b377-0de3d193ba0d 2024-08-19T14:49:55.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:49:55.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:49:55.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:49:55.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T14:49:55.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-08-19T14:49:55.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:49:55.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:49:55.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-08-19T14:49:55.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:49:55.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-08-19T14:49:55.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:49:55.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:49:55.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:49:55.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:49:55.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-08-19T14:49:55.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-08-19T14:49:55.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:49:55.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:49:55.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:49:55.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:49:55.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:49:55.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:49:55.501 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:49:55.501 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:49:55.501 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:49:55.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:49:55.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:49:55.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:49:55.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:49:55.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:49:55.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:49:55.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:49:55.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:49:55.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:49:55.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:49:55.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:49:55.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:49:55.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:49:55.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-08-19T14:49:55.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:49:55.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7c59bef0-d717-44f2-b3df-88b33605d1af 2024-08-19T14:49:55.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 7c59bef0-d717-44f2-b3df-88b33605d1af' 2024-08-19T14:49:55.504 INFO:tasks.workunit.client.0.smithi102.stdout:add osd5 7c59bef0-d717-44f2-b3df-88b33605d1af 2024-08-19T14:49:55.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:49:55.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTW8Nm19y7HhAAy4QWLwMNbJf4KV/7sI8isQ== 2024-08-19T14:49:55.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTW8Nm19y7HhAAy4QWLwMNbJf4KV/7sI8isQ=="}' 2024-08-19T14:49:55.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7c59bef0-d717-44f2-b3df-88b33605d1af -i td/osd-scrub-repair/5/new.json 2024-08-19T14:49:55.856 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:49:55.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-08-19T14:49:55.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTW8Nm19y7HhAAy4QWLwMNbJf4KV/7sI8isQ== --osd-uuid 7c59bef0-d717-44f2-b3df-88b33605d1af 2024-08-19T14:49:55.896 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:49:55.895+0000 7f331d8db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:49:55.947 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:49:55.946+0000 7f331d8db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:49:55.947 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:49:55.947+0000 7f331d8db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:49:55.948 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:49:55.947+0000 7f331d8db640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-08-19T14:49:55.948 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:49:55.947+0000 7f331d8db640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-08-19T14:49:55.948 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:49:55.947+0000 7f331d8db640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-08-19T14:49:56.038 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:49:56.037+0000 7f331d8db640 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-08-19T14:50:02.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-08-19T14:50:02.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:50:02.042 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd5 key to auth repository 2024-08-19T14:50:02.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-08-19T14:50:02.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:50:02.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-08-19T14:50:02.384 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.5 2024-08-19T14:50:02.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:50:02.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:50:02.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:50:02.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:50:02.410 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:50:02.410+0000 7fcdff0df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:50:02.412 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:50:02.412+0000 7fcdff0df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:50:02.413 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:50:02.413+0000 7fcdff0df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:50:02.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-08-19T14:50:02.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:50:02.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-08-19T14:50:02.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:50:02.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:50:02.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:02.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:50:02.680 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:50:02.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:02.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:50:02.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:02.947 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:50:02.946+0000 7fcdff0df640 -1 Falling back to public interface 2024-08-19T14:50:03.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:03.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:03.949 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:50:03.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:50:03.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:03.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:50:04.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:05.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:05.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:05.223 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:50:05.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:50:05.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:05.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:50:05.284 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:50:05.283+0000 7fcdff0df640 -1 osd.5 0 log_to_monitors true 2024-08-19T14:50:05.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:06.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:06.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:06.496 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:50:06.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:50:06.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:06.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:50:06.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:07.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:07.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:07.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:50:07.914 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:50:07.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:07.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:50:08.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:09.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:09.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:09.247 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:50:09.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:50:09.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:09.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:50:09.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:10.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:10.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:10.596 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:50:10.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:50:10.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:10.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:50:10.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:11.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:11.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:11.932 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:50:11.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:50:11.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:11.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:50:12.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:13.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:13.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:13.295 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:50:13.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:50:13.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:13.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:50:13.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:14.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:14.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:14.577 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:50:14.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:50:14.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:14.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:50:14.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:15.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:15.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:15.950 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:50:15.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:50:15.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:15.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:50:16.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:17.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:17.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:17.263 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:50:17.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:50:17.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:17.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:50:17.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:18.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:18.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:18.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:50:18.546 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:50:18.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:18.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:50:19.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:20.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:20.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:20.274 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:50:20.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:50:20.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:20.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:50:20.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:21.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:21.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:21.996 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:50:21.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:50:21.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:21.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:50:22.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:23.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:23.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:23.805 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:50:23.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:50:23.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:23.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:50:24.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:25.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:25.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:25.470 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:50:25.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:50:25.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:25.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:50:25.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:26.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:26.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:26.761 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:50:26.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:50:26.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:26.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:50:27.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:28.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:28.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:28.523 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:50:28.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:50:28.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:28.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:50:28.795 INFO:tasks.workunit.client.0.smithi102.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1430926772,v1:127.0.0.1:6843/1430926772] [v2:127.0.0.1:6844/1430926772,v1:127.0.0.1:6845/1430926772] exists,up 7c59bef0-d717-44f2-b3df-88b33605d1af 2024-08-19T14:50:28.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:50:28.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:50:28.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:50:28.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T14:50:28.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-08-19T14:50:28.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:50:28.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:50:28.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-08-19T14:50:28.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:50:28.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-08-19T14:50:28.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:50:28.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:50:28.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:50:28.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:50:28.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-08-19T14:50:28.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-08-19T14:50:28.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:50:28.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:50:28.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:50:28.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:50:28.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:50:28.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:50:28.801 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:50:28.802 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:50:28.802 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:50:28.802 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:50:28.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:50:28.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:50:28.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:50:28.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:50:28.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:50:28.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:50:28.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:50:28.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:50:28.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:50:28.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:50:28.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:50:28.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:50:28.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-08-19T14:50:28.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:50:28.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=af232db0-1f5c-47f1-a45a-a1c66cd7bd26 2024-08-19T14:50:28.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 af232db0-1f5c-47f1-a45a-a1c66cd7bd26' 2024-08-19T14:50:28.803 INFO:tasks.workunit.client.0.smithi102.stdout:add osd6 af232db0-1f5c-47f1-a45a-a1c66cd7bd26 2024-08-19T14:50:28.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:50:28.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC0W8Nm0xe5MBAAN/JvkfB1w+qqINKH3Ql7xQ== 2024-08-19T14:50:28.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC0W8Nm0xe5MBAAN/JvkfB1w+qqINKH3Ql7xQ=="}' 2024-08-19T14:50:28.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new af232db0-1f5c-47f1-a45a-a1c66cd7bd26 -i td/osd-scrub-repair/6/new.json 2024-08-19T14:50:29.160 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:50:29.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-08-19T14:50:29.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0W8Nm0xe5MBAAN/JvkfB1w+qqINKH3Ql7xQ== --osd-uuid af232db0-1f5c-47f1-a45a-a1c66cd7bd26 2024-08-19T14:50:29.200 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:50:29.199+0000 7f7e6460d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:50:29.247 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:50:29.246+0000 7f7e6460d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:50:29.247 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:50:29.247+0000 7f7e6460d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:50:29.248 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:50:29.247+0000 7f7e6460d640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-08-19T14:50:29.248 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:50:29.247+0000 7f7e6460d640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-08-19T14:50:29.248 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:50:29.247+0000 7f7e6460d640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-08-19T14:50:29.319 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:50:29.318+0000 7f7e6460d640 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-08-19T14:50:35.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-08-19T14:50:35.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:50:35.434 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd6 key to auth repository 2024-08-19T14:50:35.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-08-19T14:50:35.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:50:35.787 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.6 2024-08-19T14:50:35.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-08-19T14:50:35.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:50:35.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:50:35.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:50:35.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:50:35.814 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:50:35.814+0000 7fab0aa15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:50:35.816 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:50:35.815+0000 7fab0aa15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:50:35.817 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:50:35.817+0000 7fab0aa15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:50:36.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-08-19T14:50:36.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:50:36.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-08-19T14:50:36.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:50:36.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:50:36.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:36.057 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:50:36.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:50:36.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:36.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:50:36.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:36.338 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:50:36.338+0000 7fab0aa15640 -1 Falling back to public interface 2024-08-19T14:50:37.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:37.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:37.323 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:50:37.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:50:37.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:37.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:50:37.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:38.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:38.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:38.601 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:50:38.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:50:38.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:38.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:50:38.791 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:50:38.790+0000 7fab0aa15640 -1 osd.6 0 log_to_monitors true 2024-08-19T14:50:38.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:39.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:39.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:39.885 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:50:39.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:50:39.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:39.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:50:40.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:41.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:41.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:41.162 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:50:41.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:50:41.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:41.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:50:41.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:42.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:42.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:42.474 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:50:42.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:50:42.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:42.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:50:42.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:43.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:43.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:43.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:50:43.861 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:50:43.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:43.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:50:44.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:45.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:45.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:45.139 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:50:45.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:50:45.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:45.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:50:45.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:46.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:46.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:46.436 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:50:46.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:50:46.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:46.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:50:46.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:47.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:47.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:47.884 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:50:47.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:50:47.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:47.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:50:48.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:49.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:49.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:49.171 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:50:49.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:50:49.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:49.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:50:49.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:50.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:50.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:50.489 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:50:50.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:50:50.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:50.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:50:51.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:52.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:52.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:52.089 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:50:52.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:50:52.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:52.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:50:52.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:53.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:53.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:53.765 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:50:53.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:50:53.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:53.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:50:54.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:55.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:55.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:55.527 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:50:55.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:50:55.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:55.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:50:55.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:56.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:56.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:56.801 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:50:56.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:50:56.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:56.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:50:57.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:50:58.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:50:58.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:50:58.692 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:50:58.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:50:58.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:50:58.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:50:59.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:00.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:00.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:00.447 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:51:00.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:51:00.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:00.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:51:01.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:02.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:02.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:02.168 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:51:02.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:51:02.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:02.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:51:02.437 INFO:tasks.workunit.client.0.smithi102.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2538536557,v1:127.0.0.1:6851/2538536557] [v2:127.0.0.1:6852/2538536557,v1:127.0.0.1:6853/2538536557] exists,up af232db0-1f5c-47f1-a45a-a1c66cd7bd26 2024-08-19T14:51:02.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:51:02.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:51:02.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:51:02.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T14:51:02.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-08-19T14:51:02.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:51:02.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:51:02.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-08-19T14:51:02.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:51:02.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-08-19T14:51:02.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:51:02.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:51:02.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:51:02.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:51:02.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-08-19T14:51:02.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-08-19T14:51:02.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:51:02.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:51:02.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:51:02.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:51:02.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:51:02.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:51:02.443 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:51:02.443 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:51:02.443 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:51:02.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:51:02.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:51:02.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:51:02.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:51:02.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:51:02.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:51:02.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:51:02.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:51:02.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:51:02.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:51:02.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:51:02.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:51:02.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:51:02.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-08-19T14:51:02.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:51:02.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e98beccc-5780-43be-819a-60becee7c705 2024-08-19T14:51:02.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 e98beccc-5780-43be-819a-60becee7c705' 2024-08-19T14:51:02.445 INFO:tasks.workunit.client.0.smithi102.stdout:add osd7 e98beccc-5780-43be-819a-60becee7c705 2024-08-19T14:51:02.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:51:02.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDWW8NmNAlmGxAAGOhbCUnb2lRtmXEyELXkMw== 2024-08-19T14:51:02.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDWW8NmNAlmGxAAGOhbCUnb2lRtmXEyELXkMw=="}' 2024-08-19T14:51:02.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e98beccc-5780-43be-819a-60becee7c705 -i td/osd-scrub-repair/7/new.json 2024-08-19T14:51:02.902 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:51:02.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-08-19T14:51:02.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDWW8NmNAlmGxAAGOhbCUnb2lRtmXEyELXkMw== --osd-uuid e98beccc-5780-43be-819a-60becee7c705 2024-08-19T14:51:02.943 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:02.942+0000 7f21f432c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:51:02.997 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:02.996+0000 7f21f432c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:51:02.997 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:02.997+0000 7f21f432c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:51:02.998 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:02.997+0000 7f21f432c640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-08-19T14:51:02.998 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:02.997+0000 7f21f432c640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-08-19T14:51:02.998 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:02.997+0000 7f21f432c640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-08-19T14:51:03.113 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:03.112+0000 7f21f432c640 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-08-19T14:51:08.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-08-19T14:51:08.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:51:08.771 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd7 key to auth repository 2024-08-19T14:51:08.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-08-19T14:51:08.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:51:09.118 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.7 2024-08-19T14:51:09.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-08-19T14:51:09.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:51:09.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:51:09.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:51:09.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:51:09.145 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:09.144+0000 7fbf50432640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:51:09.147 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:09.146+0000 7fbf50432640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:51:09.148 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:09.147+0000 7fbf50432640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:51:09.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-08-19T14:51:09.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:51:09.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-08-19T14:51:09.388 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:51:09.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:51:09.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:51:09.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:09.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:51:09.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:09.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:51:09.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:09.671+0000 7fbf50432640 -1 Falling back to public interface 2024-08-19T14:51:10.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:10.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:10.656 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:51:10.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:51:10.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:10.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:51:10.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:11.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:11.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:11.931 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:51:11.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:51:11.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:11.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:51:12.009 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:12.009+0000 7fbf50432640 -1 osd.7 0 log_to_monitors true 2024-08-19T14:51:12.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:13.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:13.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:13.219 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:51:13.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:51:13.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:13.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:51:13.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:14.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:14.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:14.493 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:51:14.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:51:14.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:14.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:51:14.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:15.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:15.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:15.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:51:15.796 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:51:15.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:15.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:51:16.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:17.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:17.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:17.078 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:51:17.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:51:17.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:17.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:51:17.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:18.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:18.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:18.363 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:51:18.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:51:18.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:18.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:51:18.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:19.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:19.644 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:51:19.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:19.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:51:19.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:19.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:51:19.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:20.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:20.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:20.921 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:51:20.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:51:20.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:20.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:51:21.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:22.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:22.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:22.194 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:51:22.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:51:22.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:22.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:51:22.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:23.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:23.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:23.473 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:51:23.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:51:23.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:23.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:51:23.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:24.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:24.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:24.879 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:51:24.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:51:24.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:24.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:51:25.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:26.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:26.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:26.623 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:51:26.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:51:26.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:26.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:51:26.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:27.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:27.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:27.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:51:27.965 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:51:27.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:27.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:51:28.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:29.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:29.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:29.495 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:51:29.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:51:29.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:29.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:51:29.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:30.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:30.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:30.832 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:51:30.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:51:30.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:30.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:51:31.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:32.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:32.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:32.314 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:51:32.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:51:32.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:32.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:51:32.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:33.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:33.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:33.882 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:51:33.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:51:33.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:33.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:51:34.169 INFO:tasks.workunit.client.0.smithi102.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/2613003670,v1:127.0.0.1:6859/2613003670] [v2:127.0.0.1:6860/2613003670,v1:127.0.0.1:6861/2613003670] exists,up e98beccc-5780-43be-819a-60becee7c705 2024-08-19T14:51:34.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:51:34.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:51:34.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:51:34.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T14:51:34.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-08-19T14:51:34.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:51:34.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:51:34.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-08-19T14:51:34.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:51:34.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-08-19T14:51:34.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:51:34.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:51:34.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:51:34.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:51:34.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-08-19T14:51:34.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-08-19T14:51:34.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:51:34.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:51:34.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:51:34.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:51:34.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:51:34.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:51:34.175 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:51:34.175 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:51:34.175 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:51:34.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:51:34.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:51:34.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:51:34.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:51:34.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:51:34.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:51:34.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:51:34.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:51:34.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:51:34.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:51:34.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:51:34.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:51:34.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:51:34.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-08-19T14:51:34.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:51:34.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5d39fcc7-e8e1-4d93-a9ff-7016cb353ae1 2024-08-19T14:51:34.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 5d39fcc7-e8e1-4d93-a9ff-7016cb353ae1' 2024-08-19T14:51:34.177 INFO:tasks.workunit.client.0.smithi102.stdout:add osd8 5d39fcc7-e8e1-4d93-a9ff-7016cb353ae1 2024-08-19T14:51:34.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:51:34.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD2W8Nmdq5qCxAAkuguB7Z1Pfwx4TXSNYJeuw== 2024-08-19T14:51:34.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD2W8Nmdq5qCxAAkuguB7Z1Pfwx4TXSNYJeuw=="}' 2024-08-19T14:51:34.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5d39fcc7-e8e1-4d93-a9ff-7016cb353ae1 -i td/osd-scrub-repair/8/new.json 2024-08-19T14:51:34.510 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:51:34.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-08-19T14:51:34.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD2W8Nmdq5qCxAAkuguB7Z1Pfwx4TXSNYJeuw== --osd-uuid 5d39fcc7-e8e1-4d93-a9ff-7016cb353ae1 2024-08-19T14:51:34.552 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:34.551+0000 7f32b0a1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:51:34.578 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:34.577+0000 7f32b0a1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:51:34.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:34.578+0000 7f32b0a1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:51:34.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:34.579+0000 7f32b0a1d640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-08-19T14:51:34.580 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:34.579+0000 7f32b0a1d640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-08-19T14:51:34.580 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:34.579+0000 7f32b0a1d640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-08-19T14:51:34.645 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:34.644+0000 7f32b0a1d640 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-08-19T14:51:41.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-08-19T14:51:41.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:51:41.137 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd8 key to auth repository 2024-08-19T14:51:41.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-08-19T14:51:41.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:51:41.484 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.8 2024-08-19T14:51:41.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-08-19T14:51:41.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:51:41.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:51:41.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:51:41.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:51:41.511 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:41.511+0000 7ff56b411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:51:41.513 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:41.513+0000 7ff56b411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:51:41.515 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:41.514+0000 7ff56b411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:51:41.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-08-19T14:51:41.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:51:41.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-08-19T14:51:41.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:51:41.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:51:41.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:41.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:51:41.756 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:51:41.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:41.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:51:42.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:42.031 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:42.031+0000 7ff56b411640 -1 Falling back to public interface 2024-08-19T14:51:43.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:43.025 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:51:43.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:43.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:51:43.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:43.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:51:43.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:44.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:44.297 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:51:44.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:44.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:51:44.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:44.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:51:44.355 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:51:44.355+0000 7ff56b411640 -1 osd.8 0 log_to_monitors true 2024-08-19T14:51:44.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:45.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:45.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:45.578 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:51:45.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:51:45.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:45.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:51:45.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:46.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:46.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:46.859 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:51:46.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:51:46.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:46.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:51:47.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:48.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:48.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:48.134 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:51:48.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:51:48.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:48.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:51:48.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:49.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:49.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:49.406 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:51:49.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:51:49.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:49.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:51:49.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:50.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:50.682 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:51:50.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:50.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:51:50.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:50.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:51:50.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:51.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:51.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:51.966 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:51:51.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:51:51.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:51.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:51:52.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:53.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:53.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:53.312 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:51:53.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:51:53.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:53.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:51:53.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:54.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:54.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:54.635 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:51:54.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:51:54.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:54.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:51:54.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:55.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:55.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:55.920 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:51:55.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:51:55.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:55.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:51:56.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:57.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:57.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:57.197 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:51:57.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:51:57.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:57.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:51:57.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:51:58.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:51:58.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:51:58.915 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:51:58.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:51:58.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:51:58.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:51:59.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:00.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:00.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:00.467 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:52:00.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:52:00.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:00.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:52:01.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:02.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:02.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:02.289 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:52:02.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:52:02.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:02.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:52:02.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:03.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:03.701 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:52:03.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:03.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:52:03.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:03.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:52:04.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:05.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:05.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:05.289 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:52:05.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:52:05.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:05.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:52:05.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:06.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:06.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:06.632 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:52:06.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:52:06.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:06.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:52:07.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:08.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:08.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:08.346 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T14:52:08.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T14:52:08.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:08.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T14:52:08.651 INFO:tasks.workunit.client.0.smithi102.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/334457317,v1:127.0.0.1:6867/334457317] [v2:127.0.0.1:6868/334457317,v1:127.0.0.1:6869/334457317] exists,up 5d39fcc7-e8e1-4d93-a9ff-7016cb353ae1 2024-08-19T14:52:08.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:52:08.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:52:08.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:52:08.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T14:52:08.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-08-19T14:52:08.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:52:08.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:52:08.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-08-19T14:52:08.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:52:08.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-08-19T14:52:08.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:52:08.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:52:08.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:52:08.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:52:08.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-08-19T14:52:08.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-08-19T14:52:08.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:52:08.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:52:08.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:52:08.656 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:52:08.656 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:52:08.656 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:52:08.656 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:52:08.656 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:52:08.656 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:52:08.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:52:08.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:52:08.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:52:08.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:52:08.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:52:08.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:52:08.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:52:08.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:52:08.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:52:08.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:52:08.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:52:08.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:52:08.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:52:08.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-08-19T14:52:08.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:52:08.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d4d5322d-f118-4a96-ac94-665aa58c0de8 2024-08-19T14:52:08.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 d4d5322d-f118-4a96-ac94-665aa58c0de8' 2024-08-19T14:52:08.658 INFO:tasks.workunit.client.0.smithi102.stdout:add osd9 d4d5322d-f118-4a96-ac94-665aa58c0de8 2024-08-19T14:52:08.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:52:08.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAYXMNm7RcWKBAAT8oF3CjIkMb8zzlX5Dd7Hg== 2024-08-19T14:52:08.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAYXMNm7RcWKBAAT8oF3CjIkMb8zzlX5Dd7Hg=="}' 2024-08-19T14:52:08.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d4d5322d-f118-4a96-ac94-665aa58c0de8 -i td/osd-scrub-repair/9/new.json 2024-08-19T14:52:08.998 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:52:09.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-08-19T14:52:09.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAYXMNm7RcWKBAAT8oF3CjIkMb8zzlX5Dd7Hg== --osd-uuid d4d5322d-f118-4a96-ac94-665aa58c0de8 2024-08-19T14:52:09.039 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:52:09.038+0000 7fa12d8e5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:52:09.074 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:52:09.074+0000 7fa12d8e5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:52:09.075 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:52:09.075+0000 7fa12d8e5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:52:09.076 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:52:09.075+0000 7fa12d8e5640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-08-19T14:52:09.076 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:52:09.075+0000 7fa12d8e5640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-08-19T14:52:09.076 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:52:09.075+0000 7fa12d8e5640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-08-19T14:52:09.146 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:52:09.145+0000 7fa12d8e5640 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-08-19T14:52:14.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-08-19T14:52:14.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:52:14.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-08-19T14:52:14.658 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd9 key to auth repository 2024-08-19T14:52:14.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:52:15.014 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.9 2024-08-19T14:52:15.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-08-19T14:52:15.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:52:15.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:52:15.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:52:15.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:52:15.042 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:52:15.041+0000 7f812a81e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:52:15.043 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:52:15.043+0000 7f812a81e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:52:15.044 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:52:15.044+0000 7f812a81e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:52:15.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-08-19T14:52:15.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:52:15.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-08-19T14:52:15.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:52:15.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:52:15.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:15.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:52:15.284 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:52:15.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:15.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:15.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:15.562 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:52:15.562+0000 7f812a81e640 -1 Falling back to public interface 2024-08-19T14:52:16.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:16.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:16.552 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:52:16.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:52:16.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:16.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:16.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:17.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:17.830 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:52:17.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:17.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:52:17.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:17.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:17.881 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:52:17.880+0000 7f812a81e640 -1 osd.9 0 log_to_monitors true 2024-08-19T14:52:18.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:19.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:19.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:19.109 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:52:19.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:52:19.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:19.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:19.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:20.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:20.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:20.498 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:52:20.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:52:20.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:20.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:20.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:21.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:21.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:21.913 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:52:21.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:52:21.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:21.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:22.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:23.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:23.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:23.192 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:52:23.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:52:23.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:23.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:23.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:24.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:24.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:24.488 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:52:24.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:52:24.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:24.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:24.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:25.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:25.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:25.813 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:52:25.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:52:25.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:25.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:26.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:27.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:27.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:27.092 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:52:27.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:52:27.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:27.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:27.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:28.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:28.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:28.375 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:52:28.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:52:28.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:28.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:28.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:29.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:29.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:29.676 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:52:29.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:52:29.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:29.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:29.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:30.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:30.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:30.954 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:52:30.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:52:30.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:30.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:31.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:32.670 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:52:32.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:52:32.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:32.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:33.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:34.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:34.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:34.569 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:52:34.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:52:34.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:34.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:35.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:36.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:36.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:36.429 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:52:36.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:52:36.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:36.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:36.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:37.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:37.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:37.751 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:52:37.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:52:37.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:37.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:38.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:39.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:39.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:39.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:52:39.370 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:52:39.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:39.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:39.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:40.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:40.686 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:52:40.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:40.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:52:40.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:40.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:41.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:52:42.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:52:42.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:52:42.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T14:52:42.508 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T14:52:42.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:52:42.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:52:42.778 INFO:tasks.workunit.client.0.smithi102.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/1558617647,v1:127.0.0.1:6875/1558617647] [v2:127.0.0.1:6876/1558617647,v1:127.0.0.1:6877/1558617647] exists,up d4d5322d-f118-4a96-ac94-665aa58c0de8 2024-08-19T14:52:42.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:52:42.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:52:42.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:52:42.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-08-19T14:52:42.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T14:52:43.039 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-08-19T14:52:43.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T14:52:43.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T14:52:43.413 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' created 2024-08-19T14:52:43.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T14:52:44.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T14:52:46.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-08-19T14:52:46.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:52:46.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:52:46.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:52:46.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:52:46.848 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:52:46.848 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:52:46.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:52:46.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:52:46.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:52:46.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:52:46.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:52:46.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:52:46.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:52:46.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:52:46.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:52:47.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:52:47.243 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:52:47.243 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:52:47.243 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:52:47.243 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-08-19T14:52:47.243 INFO:tasks.workunit.client.0.smithi102.stderr:5 2024-08-19T14:52:47.243 INFO:tasks.workunit.client.0.smithi102.stderr:6 2024-08-19T14:52:47.243 INFO:tasks.workunit.client.0.smithi102.stderr:7 2024-08-19T14:52:47.243 INFO:tasks.workunit.client.0.smithi102.stderr:8 2024-08-19T14:52:47.243 INFO:tasks.workunit.client.0.smithi102.stderr:9' 2024-08-19T14:52:47.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:52:47.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:47.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:52:47.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836550 2024-08-19T14:52:47.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836550 2024-08-19T14:52:47.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550' 2024-08-19T14:52:47.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:47.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:52:47.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673020 2024-08-19T14:52:47.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673020 2024-08-19T14:52:47.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673020' 2024-08-19T14:52:47.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:47.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:52:47.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509493 2024-08-19T14:52:47.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509493 2024-08-19T14:52:47.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673020 2-64424509493' 2024-08-19T14:52:47.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:47.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:52:47.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345966 2024-08-19T14:52:47.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345966 2024-08-19T14:52:47.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673020 2-64424509493 3-85899345966' 2024-08-19T14:52:47.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:47.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T14:52:47.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182437 2024-08-19T14:52:47.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182437 2024-08-19T14:52:47.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673020 2-64424509493 3-85899345966 4-107374182437' 2024-08-19T14:52:47.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:47.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T14:52:47.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018912 2024-08-19T14:52:47.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018912 2024-08-19T14:52:47.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673020 2-64424509493 3-85899345966 4-107374182437 5-128849018912' 2024-08-19T14:52:47.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:47.862 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T14:52:47.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855388 2024-08-19T14:52:47.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855388 2024-08-19T14:52:47.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673020 2-64424509493 3-85899345966 4-107374182437 5-128849018912 6-150323855388' 2024-08-19T14:52:47.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:47.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T14:52:48.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691861 2024-08-19T14:52:48.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691861 2024-08-19T14:52:48.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673020 2-64424509493 3-85899345966 4-107374182437 5-128849018912 6-150323855388 7-171798691861' 2024-08-19T14:52:48.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:48.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T14:52:48.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528331 2024-08-19T14:52:48.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528331 2024-08-19T14:52:48.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673020 2-64424509493 3-85899345966 4-107374182437 5-128849018912 6-150323855388 7-171798691861 8-193273528331' 2024-08-19T14:52:48.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:48.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T14:52:48.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364806 2024-08-19T14:52:48.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364806 2024-08-19T14:52:48.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673020 2-64424509493 3-85899345966 4-107374182437 5-128849018912 6-150323855388 7-171798691861 8-193273528331 9-214748364806' 2024-08-19T14:52:48.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:52:48.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836550 2024-08-19T14:52:48.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:52:48.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:52:48.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836550 2024-08-19T14:52:48.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:52:48.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836550 2024-08-19T14:52:48.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-08-19T14:52:48.276 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836550 2024-08-19T14:52:48.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:52:48.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836549 -lt 21474836550 2024-08-19T14:52:48.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:52:49.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:52:49.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:52:49.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836550 -lt 21474836550 2024-08-19T14:52:49.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:52:49.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673020 2024-08-19T14:52:49.849 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:52:49.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:52:49.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673020 2024-08-19T14:52:49.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:52:49.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673020 2024-08-19T14:52:49.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673020' 2024-08-19T14:52:49.852 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673020 2024-08-19T14:52:49.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:52:50.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673020 -lt 42949673020 2024-08-19T14:52:50.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:52:50.186 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509493 2024-08-19T14:52:50.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:52:50.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:52:50.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509493 2024-08-19T14:52:50.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:52:50.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509493 2024-08-19T14:52:50.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509493' 2024-08-19T14:52:50.189 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509493 2024-08-19T14:52:50.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:52:50.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509494 -lt 64424509493 2024-08-19T14:52:50.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:52:50.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345966 2024-08-19T14:52:50.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:52:50.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:52:50.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345966 2024-08-19T14:52:50.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:52:50.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345966 2024-08-19T14:52:50.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345966' 2024-08-19T14:52:50.461 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345966 2024-08-19T14:52:50.461 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:52:50.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345966 -lt 85899345966 2024-08-19T14:52:50.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:52:50.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182437 2024-08-19T14:52:50.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:52:50.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T14:52:50.750 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182437 2024-08-19T14:52:50.750 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:52:50.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182437 2024-08-19T14:52:50.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182437' 2024-08-19T14:52:50.751 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182437 2024-08-19T14:52:50.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T14:52:51.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182438 -lt 107374182437 2024-08-19T14:52:51.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:52:51.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018912 2024-08-19T14:52:51.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:52:51.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T14:52:51.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018912 2024-08-19T14:52:51.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:52:51.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018912 2024-08-19T14:52:51.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018912' 2024-08-19T14:52:51.039 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.5 seq 128849018912 2024-08-19T14:52:51.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T14:52:51.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018913 -lt 128849018912 2024-08-19T14:52:51.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:52:51.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855388 2024-08-19T14:52:51.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:52:51.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T14:52:51.322 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855388 2024-08-19T14:52:51.322 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:52:51.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855388 2024-08-19T14:52:51.323 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.6 seq 150323855388 2024-08-19T14:52:51.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855388' 2024-08-19T14:52:51.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T14:52:51.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855388 -lt 150323855388 2024-08-19T14:52:51.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:52:51.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691861 2024-08-19T14:52:51.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:52:51.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T14:52:51.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691861 2024-08-19T14:52:51.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:52:51.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691861 2024-08-19T14:52:51.601 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.7 seq 171798691861 2024-08-19T14:52:51.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691861' 2024-08-19T14:52:51.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T14:52:51.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691862 -lt 171798691861 2024-08-19T14:52:51.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:52:51.922 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528331 2024-08-19T14:52:51.922 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:52:51.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T14:52:51.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528331 2024-08-19T14:52:51.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:52:51.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528331 2024-08-19T14:52:51.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528331' 2024-08-19T14:52:51.925 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.8 seq 193273528331 2024-08-19T14:52:51.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T14:52:52.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528331 -lt 193273528331 2024-08-19T14:52:52.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:52:52.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364806 2024-08-19T14:52:52.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:52:52.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T14:52:52.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364806 2024-08-19T14:52:52.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:52:52.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364806 2024-08-19T14:52:52.236 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.9 seq 214748364806 2024-08-19T14:52:52.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364806' 2024-08-19T14:52:52.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T14:52:52.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364806 -lt 214748364806 2024-08-19T14:52:52.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:52:52.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:52:52.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:52:52.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T14:52:52.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:52:52.841 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:52:52.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:52:52.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:52:52.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:52:52.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:52:52.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:52:53.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:52:53.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:52:53.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:52:53.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:52:53.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T14:52:53.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:52:53.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:52:53.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2024-08-19T14:52:53.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-08-19T14:52:53.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-08-19T14:52:53.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-08-19T14:52:53.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-08-19T14:52:53.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-08-19T14:52:53.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-08-19T14:52:53.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-08-19T14:52:54.518 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'ecpool' created 2024-08-19T14:52:54.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T14:52:55.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-08-19T14:52:55.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-08-19T14:52:55.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:52:55.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:52:55.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:52:55.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:52:55.533 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:52:55.533 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:52:55.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:52:55.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:52:55.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:52:55.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:52:55.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:52:55.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:52:55.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:52:55.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:52:55.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:52:55.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:52:55.910 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:52:55.910 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:52:55.910 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:52:55.910 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-08-19T14:52:55.910 INFO:tasks.workunit.client.0.smithi102.stderr:5 2024-08-19T14:52:55.910 INFO:tasks.workunit.client.0.smithi102.stderr:6 2024-08-19T14:52:55.910 INFO:tasks.workunit.client.0.smithi102.stderr:7 2024-08-19T14:52:55.910 INFO:tasks.workunit.client.0.smithi102.stderr:8 2024-08-19T14:52:55.911 INFO:tasks.workunit.client.0.smithi102.stderr:9' 2024-08-19T14:52:55.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:52:55.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:55.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:52:56.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836553 2024-08-19T14:52:56.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836553 2024-08-19T14:52:56.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553' 2024-08-19T14:52:56.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:56.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:52:56.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673022 2024-08-19T14:52:56.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673022 2024-08-19T14:52:56.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673022' 2024-08-19T14:52:56.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:56.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:52:56.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509496 2024-08-19T14:52:56.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509496 2024-08-19T14:52:56.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673022 2-64424509496' 2024-08-19T14:52:56.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:56.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:52:56.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345968 2024-08-19T14:52:56.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345968 2024-08-19T14:52:56.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673022 2-64424509496 3-85899345968' 2024-08-19T14:52:56.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:56.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T14:52:56.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182440 2024-08-19T14:52:56.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182440 2024-08-19T14:52:56.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673022 2-64424509496 3-85899345968 4-107374182440' 2024-08-19T14:52:56.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:56.590 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T14:52:56.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018915 2024-08-19T14:52:56.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018915 2024-08-19T14:52:56.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673022 2-64424509496 3-85899345968 4-107374182440 5-128849018915' 2024-08-19T14:52:56.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:56.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T14:52:56.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855390 2024-08-19T14:52:56.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855390 2024-08-19T14:52:56.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673022 2-64424509496 3-85899345968 4-107374182440 5-128849018915 6-150323855390' 2024-08-19T14:52:56.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:56.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T14:52:56.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691864 2024-08-19T14:52:56.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691864 2024-08-19T14:52:56.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673022 2-64424509496 3-85899345968 4-107374182440 5-128849018915 6-150323855390 7-171798691864' 2024-08-19T14:52:56.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:56.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T14:52:57.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528333 2024-08-19T14:52:57.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528333 2024-08-19T14:52:57.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673022 2-64424509496 3-85899345968 4-107374182440 5-128849018915 6-150323855390 7-171798691864 8-193273528333' 2024-08-19T14:52:57.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:52:57.058 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T14:52:57.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364808 2024-08-19T14:52:57.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364808 2024-08-19T14:52:57.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673022 2-64424509496 3-85899345968 4-107374182440 5-128849018915 6-150323855390 7-171798691864 8-193273528333 9-214748364808' 2024-08-19T14:52:57.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:52:57.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836553 2024-08-19T14:52:57.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:52:57.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:52:57.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836553 2024-08-19T14:52:57.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:52:57.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836553 2024-08-19T14:52:57.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2024-08-19T14:52:57.165 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836553 2024-08-19T14:52:57.165 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:52:57.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836553 -lt 21474836553 2024-08-19T14:52:57.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:52:57.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673022 2024-08-19T14:52:57.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:52:57.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:52:57.433 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673022 2024-08-19T14:52:57.433 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:52:57.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673022 2024-08-19T14:52:57.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673022' 2024-08-19T14:52:57.435 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673022 2024-08-19T14:52:57.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:52:57.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673023 -lt 42949673022 2024-08-19T14:52:57.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:52:57.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509496 2024-08-19T14:52:57.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:52:57.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:52:57.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509496 2024-08-19T14:52:57.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:52:57.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509496 2024-08-19T14:52:57.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509496' 2024-08-19T14:52:57.709 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509496 2024-08-19T14:52:57.709 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:52:57.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509496 -lt 64424509496 2024-08-19T14:52:57.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:52:57.979 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345968 2024-08-19T14:52:57.979 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:52:57.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:52:57.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345968 2024-08-19T14:52:57.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:52:57.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345968 2024-08-19T14:52:57.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345968' 2024-08-19T14:52:57.981 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345968 2024-08-19T14:52:57.982 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:52:58.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345968 -lt 85899345968 2024-08-19T14:52:58.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:52:58.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182440 2024-08-19T14:52:58.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:52:58.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T14:52:58.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182440 2024-08-19T14:52:58.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:52:58.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182440 2024-08-19T14:52:58.251 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182440 2024-08-19T14:52:58.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182440' 2024-08-19T14:52:58.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T14:52:58.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182439 -lt 107374182440 2024-08-19T14:52:58.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:52:59.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:52:59.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T14:52:59.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182440 -lt 107374182440 2024-08-19T14:52:59.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:52:59.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018915 2024-08-19T14:52:59.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:52:59.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T14:52:59.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018915 2024-08-19T14:52:59.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:52:59.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018915 2024-08-19T14:52:59.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018915' 2024-08-19T14:52:59.795 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.5 seq 128849018915 2024-08-19T14:52:59.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T14:53:00.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018915 -lt 128849018915 2024-08-19T14:53:00.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:00.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855390 2024-08-19T14:53:00.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:00.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T14:53:00.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855390 2024-08-19T14:53:00.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:00.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855390 2024-08-19T14:53:00.066 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.6 seq 150323855390 2024-08-19T14:53:00.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855390' 2024-08-19T14:53:00.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T14:53:00.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855391 -lt 150323855390 2024-08-19T14:53:00.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:00.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691864 2024-08-19T14:53:00.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:00.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T14:53:00.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691864 2024-08-19T14:53:00.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:00.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691864 2024-08-19T14:53:00.334 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.7 seq 171798691864 2024-08-19T14:53:00.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691864' 2024-08-19T14:53:00.334 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T14:53:00.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691865 -lt 171798691864 2024-08-19T14:53:00.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:00.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528333 2024-08-19T14:53:00.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:00.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T14:53:00.605 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528333 2024-08-19T14:53:00.605 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:00.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528333 2024-08-19T14:53:00.606 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.8 seq 193273528333 2024-08-19T14:53:00.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528333' 2024-08-19T14:53:00.606 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T14:53:00.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528334 -lt 193273528333 2024-08-19T14:53:00.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:00.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364808 2024-08-19T14:53:00.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:00.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T14:53:00.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364808 2024-08-19T14:53:00.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:00.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364808 2024-08-19T14:53:00.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364808' 2024-08-19T14:53:00.903 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.9 seq 214748364808 2024-08-19T14:53:00.903 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T14:53:01.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364809 -lt 214748364808 2024-08-19T14:53:01.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:53:01.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:53:01.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:53:01.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:53:01.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:53:01.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:53:01.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:53:01.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:53:01.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:53:01.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:53:01.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:53:01.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:53:01.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:53:01.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:53:01.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:53:02.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:53:02.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:53:02.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:53:02.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-08-19T14:53:02.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-08-19T14:53:02.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-08-19T14:53:02.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-08-19T14:53:02.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-08-19T14:53:02.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T14:53:02.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T14:53:02.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-08-19T14:53:02.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T14:53:02.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T14:53:02.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T14:53:02.501 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T14:53:02.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T14:53:03.064 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T14:53:03.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T14:53:03.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T14:53:03.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-08-19T14:53:03.522 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-08-19T14:53:03.522 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-08-19T14:53:03.522 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T14:53:03.522 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:53:03.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T14:53:03.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-08-19T14:53:03.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-08-19T14:53:03.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-08-19T14:53:03.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-08-19T14:53:03.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-08-19T14:53:03.794 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:53:03.794 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-19T14:53:04.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-08-19T14:53:04.064 INFO:tasks.workunit.client.0.smithi102.stderr:5 2024-08-19T14:53:04.064 INFO:tasks.workunit.client.0.smithi102.stderr:9 2024-08-19T14:53:04.064 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:53:04.064 INFO:tasks.workunit.client.0.smithi102.stderr:6 2024-08-19T14:53:04.064 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:53:04.064 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:53:04.064 INFO:tasks.workunit.client.0.smithi102.stderr:7' 2024-08-19T14:53:04.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-08-19T14:53:04.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-08-19T14:53:04.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-08-19T14:53:04.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-08-19T14:53:04.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-08-19T14:53:04.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-08-19T14:53:04.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-08-19T14:53:04.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-08-19T14:53:04.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-08-19T14:53:04.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:53:04.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:53:04.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:53:04.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-08-19T14:53:04.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:53:04.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:53:04.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:53:04.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:53:04.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-08-19T14:53:04.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:53:04.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-08-19T14:53:04.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:53:04.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:53:04.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:53:04.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:53:04.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:53:04.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:53:04.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:53:04.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:53:04.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:53:04.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-08-19T14:53:04.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:53:04.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:53:04.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-08-19T14:53:06.064 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-08-19T14:53:06.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-08-19T14:53:06.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:53:06.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:53:06.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-19T14:53:06.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:53:06.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:53:06.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:53:06.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:53:06.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:53:06.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:53:06.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T14:53:06.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T14:53:06.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:53:06.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:53:06.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:53:06.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:53:06.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:53:06.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:53:06.585 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:53:06.585 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:53:06.586 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:53:06.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:53:06.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:53:06.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:53:06.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:53:06.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:53:06.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:53:06.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:53:06.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:53:06.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:53:06.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:53:06.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:53:06.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T14:53:06.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-19T14:53:06.587 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.3 2024-08-19T14:53:06.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:53:06.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-08-19T14:53:06.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-19T14:53:06.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:53:06.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:53:06.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:53:06.613 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:53:06.613+0000 7ff8ec217640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:53:06.616 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:53:06.615+0000 7ff8ec217640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:53:06.617 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:53:06.616+0000 7ff8ec217640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:53:06.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-19T14:53:06.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:53:06.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T14:53:06.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:53:06.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:53:06.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:53:06.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:53:06.864 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:53:06.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:53:06.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:53:07.137 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:53:07.136+0000 7ff8ec217640 -1 Falling back to public interface 2024-08-19T14:53:07.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:53:08.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:53:08.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:53:08.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:53:08.175 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:53:08.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:53:08.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:53:08.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:53:09.414 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:53:09.413+0000 7ff8ec217640 -1 osd.3 66 log_to_monitors true 2024-08-19T14:53:09.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:53:09.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:53:09.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:53:09.449 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:53:09.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:53:09.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:53:09.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:53:10.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:53:10.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:53:10.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:53:10.830 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:53:10.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:53:10.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:53:11.221 INFO:tasks.workunit.client.0.smithi102.stderr:osd.3 up in weight 1 up_from 70 up_thru 70 down_at 67 last_clean_interval [20,66) [v2:127.0.0.1:6826/31107394,v1:127.0.0.1:6827/31107394] [v2:127.0.0.1:6828/31107394,v1:127.0.0.1:6829/31107394] exists,up 0a924bc7-1b05-4420-8781-adc75bdf85fc 2024-08-19T14:53:11.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:53:11.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:53:11.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:53:11.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:53:11.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:53:11.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:53:11.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:53:11.221 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:53:11.222 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:53:11.222 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:53:11.222 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:53:11.222 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:53:11.222 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:53:11.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:53:11.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:53:11.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:53:11.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:53:11.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:53:11.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:53:11.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:53:11.586 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:53:11.586 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:53:11.586 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:53:11.587 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-08-19T14:53:11.587 INFO:tasks.workunit.client.0.smithi102.stderr:5 2024-08-19T14:53:11.587 INFO:tasks.workunit.client.0.smithi102.stderr:6 2024-08-19T14:53:11.587 INFO:tasks.workunit.client.0.smithi102.stderr:7 2024-08-19T14:53:11.587 INFO:tasks.workunit.client.0.smithi102.stderr:8 2024-08-19T14:53:11.587 INFO:tasks.workunit.client.0.smithi102.stderr:9' 2024-08-19T14:53:11.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:53:11.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:11.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:53:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836557 2024-08-19T14:53:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836557 2024-08-19T14:53:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557' 2024-08-19T14:53:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:11.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:53:11.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673027 2024-08-19T14:53:11.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673027 2024-08-19T14:53:11.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673027' 2024-08-19T14:53:11.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:11.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:53:11.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509500 2024-08-19T14:53:11.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509500 2024-08-19T14:53:11.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673027 2-64424509500' 2024-08-19T14:53:11.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:11.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:53:12.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710722 2024-08-19T14:53:12.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710722 2024-08-19T14:53:12.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673027 2-64424509500 3-300647710722' 2024-08-19T14:53:12.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:12.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T14:53:12.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182444 2024-08-19T14:53:12.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182444 2024-08-19T14:53:12.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673027 2-64424509500 3-300647710722 4-107374182444' 2024-08-19T14:53:12.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:12.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T14:53:12.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018919 2024-08-19T14:53:12.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018919 2024-08-19T14:53:12.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673027 2-64424509500 3-300647710722 4-107374182444 5-128849018919' 2024-08-19T14:53:12.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:12.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T14:53:12.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855394 2024-08-19T14:53:12.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855394 2024-08-19T14:53:12.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673027 2-64424509500 3-300647710722 4-107374182444 5-128849018919 6-150323855394' 2024-08-19T14:53:12.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:12.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T14:53:12.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691868 2024-08-19T14:53:12.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691868 2024-08-19T14:53:12.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673027 2-64424509500 3-300647710722 4-107374182444 5-128849018919 6-150323855394 7-171798691868' 2024-08-19T14:53:12.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:12.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T14:53:12.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528337 2024-08-19T14:53:12.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528337 2024-08-19T14:53:12.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673027 2-64424509500 3-300647710722 4-107374182444 5-128849018919 6-150323855394 7-171798691868 8-193273528337' 2024-08-19T14:53:12.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:12.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T14:53:12.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364813 2024-08-19T14:53:12.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364813 2024-08-19T14:53:12.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673027 2-64424509500 3-300647710722 4-107374182444 5-128849018919 6-150323855394 7-171798691868 8-193273528337 9-214748364813' 2024-08-19T14:53:12.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:12.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836557 2024-08-19T14:53:12.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:12.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:53:12.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836557 2024-08-19T14:53:12.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:12.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836557 2024-08-19T14:53:12.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836557' 2024-08-19T14:53:12.630 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836557 2024-08-19T14:53:12.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:53:12.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836557 -lt 21474836557 2024-08-19T14:53:12.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:12.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673027 2024-08-19T14:53:12.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:12.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:53:12.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673027 2024-08-19T14:53:12.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:12.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673027 2024-08-19T14:53:12.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673027' 2024-08-19T14:53:12.903 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949673027 2024-08-19T14:53:12.903 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:53:13.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673027 -lt 42949673027 2024-08-19T14:53:13.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:13.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509500 2024-08-19T14:53:13.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:13.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:53:13.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509500 2024-08-19T14:53:13.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:13.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509500 2024-08-19T14:53:13.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509500' 2024-08-19T14:53:13.176 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509500 2024-08-19T14:53:13.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:53:13.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509500 -lt 64424509500 2024-08-19T14:53:13.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:13.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710722 2024-08-19T14:53:13.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:13.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:53:13.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710722 2024-08-19T14:53:13.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:13.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710722 2024-08-19T14:53:13.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710722' 2024-08-19T14:53:13.447 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.3 seq 300647710722 2024-08-19T14:53:13.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:53:13.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710722 -lt 300647710722 2024-08-19T14:53:13.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:13.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182444 2024-08-19T14:53:13.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:13.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T14:53:13.717 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182444 2024-08-19T14:53:13.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:13.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182444 2024-08-19T14:53:13.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182444' 2024-08-19T14:53:13.719 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.4 seq 107374182444 2024-08-19T14:53:13.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T14:53:13.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182444 -lt 107374182444 2024-08-19T14:53:13.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:13.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018919 2024-08-19T14:53:13.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:13.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T14:53:13.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018919 2024-08-19T14:53:13.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:13.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018919 2024-08-19T14:53:13.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018919' 2024-08-19T14:53:13.994 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.5 seq 128849018919 2024-08-19T14:53:13.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T14:53:14.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018919 -lt 128849018919 2024-08-19T14:53:14.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:14.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855394 2024-08-19T14:53:14.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:14.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T14:53:14.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855394 2024-08-19T14:53:14.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:14.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855394 2024-08-19T14:53:14.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855394' 2024-08-19T14:53:14.266 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.6 seq 150323855394 2024-08-19T14:53:14.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T14:53:14.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855395 -lt 150323855394 2024-08-19T14:53:14.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:14.536 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691868 2024-08-19T14:53:14.536 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:14.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T14:53:14.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691868 2024-08-19T14:53:14.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:14.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691868 2024-08-19T14:53:14.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691868' 2024-08-19T14:53:14.538 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.7 seq 171798691868 2024-08-19T14:53:14.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T14:53:14.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691868 -lt 171798691868 2024-08-19T14:53:14.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:14.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528337 2024-08-19T14:53:14.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:14.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T14:53:14.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528337 2024-08-19T14:53:14.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:14.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528337 2024-08-19T14:53:14.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528337' 2024-08-19T14:53:14.849 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.8 seq 193273528337 2024-08-19T14:53:14.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T14:53:15.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528338 -lt 193273528337 2024-08-19T14:53:15.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:15.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364813 2024-08-19T14:53:15.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:15.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T14:53:15.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364813 2024-08-19T14:53:15.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:15.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364813 2024-08-19T14:53:15.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364813' 2024-08-19T14:53:15.123 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.9 seq 214748364813 2024-08-19T14:53:15.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T14:53:15.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364813 -lt 214748364813 2024-08-19T14:53:15.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:53:15.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:53:15.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:53:15.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:53:15.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:53:15.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:53:15.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:53:15.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:53:15.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:53:15.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:53:15.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:53:15.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:53:15.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:53:15.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:53:15.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:53:16.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:53:16.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:53:16.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:53:16.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-08-19T14:53:16.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T14:53:16.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T14:53:16.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:53:16.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:53:16.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-08-19T14:53:16.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-08-19T14:53:16.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-08-19T14:53:16.595 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-08-19T14:53:16.595 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:53:16.595 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:53:16.595 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:53:16.595 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:53:16.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T14:52:54.391806+0000 2024-08-19T14:53:16.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-08-19T14:53:17.112 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-08-19T14:53:17.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-08-19T14:52:54.391806+0000 2024-08-19T14:53:17.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T14:53:17.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:52:54.391806+0000 2024-08-19T14:53:17.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:53:17.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:53:17.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:53:17.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:53:17.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:53:17.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:53:17.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:53:17.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:53:17.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:52:54.391806+0000 '>' 2024-08-19T14:52:54.391806+0000 2024-08-19T14:53:17.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:53:18.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:53:18.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:53:18.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:53:18.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:53:18.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:53:18.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:53:18.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:53:18.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:52:54.391806+0000 '>' 2024-08-19T14:52:54.391806+0000 2024-08-19T14:53:18.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:53:19.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:53:19.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:53:19.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:53:19.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:53:19.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:53:19.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:53:19.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:53:19.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:52:54.391806+0000 '>' 2024-08-19T14:52:54.391806+0000 2024-08-19T14:53:19.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:53:20.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:53:20.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:53:20.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:53:20.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:53:20.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:53:20.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:53:20.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:53:21.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:53:17.431048+0000 '>' 2024-08-19T14:52:54.391806+0000 2024-08-19T14:53:21.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:53:21.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T14:53:21.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:53:21.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:53:21.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-08-19T14:53:21.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:53:21.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T14:53:21.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:53:21.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:53:21.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-08-19T14:53:21.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:53:21.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-08-19T14:53:21.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:53:21.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:53:21.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:53:21.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:53:21.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:53:21.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:53:21.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T14:53:21.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:53:21.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:53:21.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-08-19T14:53:21.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:53:21.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-08-19T14:53:21.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-08-19T14:53:23.238 INFO:tasks.workunit.client.0.smithi102.stdout:_ 2024-08-19T14:53:23.238 INFO:tasks.workunit.client.0.smithi102.stdout:hinfo_key 2024-08-19T14:53:23.239 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 2024-08-19T14:53:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-08-19T14:53:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:53:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:53:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-08-19T14:53:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:53:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-08-19T14:53:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:53:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:53:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:53:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:53:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-08-19T14:53:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-08-19T14:53:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:53:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:53:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:53:23.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:53:23.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:53:23.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:53:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:53:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:53:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:53:23.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:53:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:53:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:53:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:53:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:53:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:53:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:53:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:53:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:53:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:53:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:53:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-08-19T14:53:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-08-19T14:53:23.519 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.9 2024-08-19T14:53:23.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:53:23.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-08-19T14:53:23.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-08-19T14:53:23.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:53:23.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:53:23.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:53:23.544 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:53:23.544+0000 7fa9f8e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:53:23.547 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:53:23.546+0000 7fa9f8e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:53:23.548 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:53:23.547+0000 7fa9f8e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:53:23.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-08-19T14:53:23.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:53:23.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-08-19T14:53:23.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:53:23.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:53:23.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:53:23.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:53:23.794 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:53:23.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:53:23.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:53:24.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:53:24.071 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:53:24.071+0000 7fa9f8e0d640 -1 Falling back to public interface 2024-08-19T14:53:25.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:53:25.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:53:25.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:53:25.064 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:53:25.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:53:25.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:53:25.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:53:26.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:53:26.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:53:26.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:53:26.340 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:53:26.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:53:26.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:53:26.410 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:53:26.409+0000 7fa9f8e0d640 -1 osd.9 71 log_to_monitors true 2024-08-19T14:53:26.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:53:27.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:53:27.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:53:27.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:53:27.634 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:53:27.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:53:27.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:53:28.155 INFO:tasks.workunit.client.0.smithi102.stderr:osd.9 up in weight 1 up_from 75 up_thru 0 down_at 72 last_clean_interval [50,71) [v2:127.0.0.1:6874/3188335795,v1:127.0.0.1:6875/3188335795] [v2:127.0.0.1:6876/3188335795,v1:127.0.0.1:6877/3188335795] exists,up d4d5322d-f118-4a96-ac94-665aa58c0de8 2024-08-19T14:53:28.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:53:28.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:53:28.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:53:28.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:53:28.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:53:28.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:53:28.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:53:28.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:53:28.156 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:53:28.156 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:53:28.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:53:28.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:53:28.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:53:28.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:53:28.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:53:28.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:53:28.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:53:28.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:53:28.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:53:28.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:53:28.576 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:53:28.576 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:53:28.576 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:53:28.576 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-08-19T14:53:28.576 INFO:tasks.workunit.client.0.smithi102.stderr:5 2024-08-19T14:53:28.576 INFO:tasks.workunit.client.0.smithi102.stderr:6 2024-08-19T14:53:28.576 INFO:tasks.workunit.client.0.smithi102.stderr:7 2024-08-19T14:53:28.576 INFO:tasks.workunit.client.0.smithi102.stderr:8 2024-08-19T14:53:28.576 INFO:tasks.workunit.client.0.smithi102.stderr:9' 2024-08-19T14:53:28.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:53:28.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:28.577 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:53:28.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836561 2024-08-19T14:53:28.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836561 2024-08-19T14:53:28.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561' 2024-08-19T14:53:28.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:28.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:53:28.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673031 2024-08-19T14:53:28.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673031 2024-08-19T14:53:28.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673031' 2024-08-19T14:53:28.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:28.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:53:28.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509505 2024-08-19T14:53:28.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509505 2024-08-19T14:53:28.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673031 2-64424509505' 2024-08-19T14:53:28.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:28.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:53:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710726 2024-08-19T14:53:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710726 2024-08-19T14:53:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673031 2-64424509505 3-300647710726' 2024-08-19T14:53:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:29.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T14:53:29.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182448 2024-08-19T14:53:29.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182448 2024-08-19T14:53:29.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673031 2-64424509505 3-300647710726 4-107374182448' 2024-08-19T14:53:29.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:29.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T14:53:29.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018924 2024-08-19T14:53:29.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018924 2024-08-19T14:53:29.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673031 2-64424509505 3-300647710726 4-107374182448 5-128849018924' 2024-08-19T14:53:29.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:29.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T14:53:29.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855399 2024-08-19T14:53:29.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855399 2024-08-19T14:53:29.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673031 2-64424509505 3-300647710726 4-107374182448 5-128849018924 6-150323855399' 2024-08-19T14:53:29.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:29.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T14:53:29.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691872 2024-08-19T14:53:29.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691872 2024-08-19T14:53:29.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673031 2-64424509505 3-300647710726 4-107374182448 5-128849018924 6-150323855399 7-171798691872' 2024-08-19T14:53:29.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:29.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T14:53:29.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528342 2024-08-19T14:53:29.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528342 2024-08-19T14:53:29.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673031 2-64424509505 3-300647710726 4-107374182448 5-128849018924 6-150323855399 7-171798691872 8-193273528342' 2024-08-19T14:53:29.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:29.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T14:53:29.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547202 2024-08-19T14:53:29.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547202 2024-08-19T14:53:29.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673031 2-64424509505 3-300647710726 4-107374182448 5-128849018924 6-150323855399 7-171798691872 8-193273528342 9-322122547202' 2024-08-19T14:53:29.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:29.643 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836561 2024-08-19T14:53:29.643 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:29.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:53:29.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836561 2024-08-19T14:53:29.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:29.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836561 2024-08-19T14:53:29.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836561' 2024-08-19T14:53:29.645 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836561 2024-08-19T14:53:29.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:53:29.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836560 -lt 21474836561 2024-08-19T14:53:29.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:53:30.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:53:30.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:53:31.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836562 -lt 21474836561 2024-08-19T14:53:31.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:31.184 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673031 2024-08-19T14:53:31.184 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:31.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:53:31.185 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673031 2024-08-19T14:53:31.185 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:31.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673031 2024-08-19T14:53:31.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673031' 2024-08-19T14:53:31.186 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949673031 2024-08-19T14:53:31.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:53:31.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673031 -lt 42949673031 2024-08-19T14:53:31.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:31.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509505 2024-08-19T14:53:31.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:31.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:53:31.456 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509505 2024-08-19T14:53:31.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:31.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509505 2024-08-19T14:53:31.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509505' 2024-08-19T14:53:31.458 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509505 2024-08-19T14:53:31.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:53:31.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509505 -lt 64424509505 2024-08-19T14:53:31.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:31.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710726 2024-08-19T14:53:31.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:31.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:53:31.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710726 2024-08-19T14:53:31.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:31.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710726 2024-08-19T14:53:31.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710726' 2024-08-19T14:53:31.735 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.3 seq 300647710726 2024-08-19T14:53:31.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:53:32.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710727 -lt 300647710726 2024-08-19T14:53:32.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:32.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182448 2024-08-19T14:53:32.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:32.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T14:53:32.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182448 2024-08-19T14:53:32.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:32.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182448 2024-08-19T14:53:32.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182448' 2024-08-19T14:53:32.004 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.4 seq 107374182448 2024-08-19T14:53:32.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T14:53:32.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182449 -lt 107374182448 2024-08-19T14:53:32.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:32.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018924 2024-08-19T14:53:32.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:32.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T14:53:32.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018924 2024-08-19T14:53:32.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:32.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018924 2024-08-19T14:53:32.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018924' 2024-08-19T14:53:32.278 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.5 seq 128849018924 2024-08-19T14:53:32.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T14:53:32.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018924 -lt 128849018924 2024-08-19T14:53:32.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:32.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855399 2024-08-19T14:53:32.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:32.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T14:53:32.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855399 2024-08-19T14:53:32.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:32.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855399 2024-08-19T14:53:32.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855399' 2024-08-19T14:53:32.639 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.6 seq 150323855399 2024-08-19T14:53:32.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T14:53:32.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855400 -lt 150323855399 2024-08-19T14:53:32.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:32.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691872 2024-08-19T14:53:32.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:32.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T14:53:32.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691872 2024-08-19T14:53:32.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:32.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691872 2024-08-19T14:53:32.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691872' 2024-08-19T14:53:32.911 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.7 seq 171798691872 2024-08-19T14:53:32.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T14:53:33.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691873 -lt 171798691872 2024-08-19T14:53:33.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:33.180 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528342 2024-08-19T14:53:33.180 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:33.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T14:53:33.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528342 2024-08-19T14:53:33.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:33.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528342 2024-08-19T14:53:33.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528342' 2024-08-19T14:53:33.182 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.8 seq 193273528342 2024-08-19T14:53:33.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T14:53:33.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528342 -lt 193273528342 2024-08-19T14:53:33.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:33.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-322122547202 2024-08-19T14:53:33.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:33.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T14:53:33.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-322122547202 2024-08-19T14:53:33.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:33.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547202 2024-08-19T14:53:33.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 322122547202' 2024-08-19T14:53:33.454 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.9 seq 322122547202 2024-08-19T14:53:33.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T14:53:33.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547203 -lt 322122547202 2024-08-19T14:53:33.723 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:53:33.723 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:53:33.723 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:53:34.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:53:34.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:53:34.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:53:34.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:53:34.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:53:34.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:53:34.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:53:34.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:53:34.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:53:34.319 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:53:34.319 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:53:34.319 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:53:34.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:53:34.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:53:34.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:53:34.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T14:53:34.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T14:53:34.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-08-19T14:53:34.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-08-19T14:53:34.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-08-19T14:53:34.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-08-19T14:53:34.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T14:53:34.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:53:34.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:53:34.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-08-19T14:53:34.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:53:34.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T14:53:34.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:53:34.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:53:34.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-08-19T14:53:34.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:53:34.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-08-19T14:53:34.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:53:34.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:53:34.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:53:34.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:53:34.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:53:35.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:53:35.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T14:53:35.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:53:35.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:53:35.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-08-19T14:53:35.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:53:35.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-08-19T14:53:35.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-08-19T14:53:36.973 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-08-19T14:53:37.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-08-19T14:53:37.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:53:37.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:53:37.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-08-19T14:53:37.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:53:37.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-08-19T14:53:37.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:53:37.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:53:37.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:53:37.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:53:37.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-08-19T14:53:37.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-08-19T14:53:37.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:53:37.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:53:37.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:53:37.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:53:37.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:53:37.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:53:37.498 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:53:37.498 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:53:37.498 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:53:37.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:53:37.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:53:37.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:53:37.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:53:37.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:53:37.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:53:37.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:53:37.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:53:37.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:53:37.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:53:37.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:53:37.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-08-19T14:53:37.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-08-19T14:53:37.500 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.5 2024-08-19T14:53:37.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:53:37.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-08-19T14:53:37.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-08-19T14:53:37.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:53:37.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:53:37.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:53:37.526 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:53:37.525+0000 7f98ad4e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:53:37.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:53:37.527+0000 7f98ad4e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:53:37.529 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:53:37.529+0000 7f98ad4e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:53:37.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-08-19T14:53:37.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:53:37.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-08-19T14:53:37.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:53:37.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:53:37.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:53:37.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:53:37.774 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:53:37.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:53:37.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:53:38.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:53:38.058 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:53:38.057+0000 7f98ad4e8640 -1 Falling back to public interface 2024-08-19T14:53:39.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:53:39.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:53:39.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:53:39.041 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:53:39.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:53:39.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:53:39.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:53:40.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:53:40.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:53:40.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:53:40.314 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:53:40.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:53:40.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:53:40.441 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:53:40.440+0000 7f98ad4e8640 -1 osd.5 76 log_to_monitors true 2024-08-19T14:53:40.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:53:41.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:53:41.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:53:41.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:53:41.655 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:53:41.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:53:41.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:53:42.057 INFO:tasks.workunit.client.0.smithi102.stderr:osd.5 up in weight 1 up_from 81 up_thru 81 down_at 77 last_clean_interval [30,76) [v2:127.0.0.1:6842/290008158,v1:127.0.0.1:6843/290008158] [v2:127.0.0.1:6844/290008158,v1:127.0.0.1:6845/290008158] exists,up 7c59bef0-d717-44f2-b3df-88b33605d1af 2024-08-19T14:53:42.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:53:42.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:53:42.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:53:42.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:53:42.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:53:42.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:53:42.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:53:42.058 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:53:42.059 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:53:42.059 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:53:42.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:53:42.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:53:42.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:53:42.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:53:42.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:53:42.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:53:42.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:53:42.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:53:42.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:53:42.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:53:42.529 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:53:42.529 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:53:42.529 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:53:42.529 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-08-19T14:53:42.529 INFO:tasks.workunit.client.0.smithi102.stderr:5 2024-08-19T14:53:42.529 INFO:tasks.workunit.client.0.smithi102.stderr:6 2024-08-19T14:53:42.529 INFO:tasks.workunit.client.0.smithi102.stderr:7 2024-08-19T14:53:42.529 INFO:tasks.workunit.client.0.smithi102.stderr:8 2024-08-19T14:53:42.530 INFO:tasks.workunit.client.0.smithi102.stderr:9' 2024-08-19T14:53:42.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:53:42.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:42.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:53:42.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836565 2024-08-19T14:53:42.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836565 2024-08-19T14:53:42.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565' 2024-08-19T14:53:42.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:42.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:53:42.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673035 2024-08-19T14:53:42.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673035 2024-08-19T14:53:42.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673035' 2024-08-19T14:53:42.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:42.738 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:53:42.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509508 2024-08-19T14:53:42.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509508 2024-08-19T14:53:42.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673035 2-64424509508' 2024-08-19T14:53:42.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:42.979 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:53:43.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710730 2024-08-19T14:53:43.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710730 2024-08-19T14:53:43.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673035 2-64424509508 3-300647710730' 2024-08-19T14:53:43.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:43.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T14:53:43.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182452 2024-08-19T14:53:43.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182452 2024-08-19T14:53:43.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673035 2-64424509508 3-300647710730 4-107374182452' 2024-08-19T14:53:43.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:43.217 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T14:53:43.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350978 2024-08-19T14:53:43.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350978 2024-08-19T14:53:43.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673035 2-64424509508 3-300647710730 4-107374182452 5-347892350978' 2024-08-19T14:53:43.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:43.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T14:53:43.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855403 2024-08-19T14:53:43.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855403 2024-08-19T14:53:43.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673035 2-64424509508 3-300647710730 4-107374182452 5-347892350978 6-150323855403' 2024-08-19T14:53:43.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:43.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T14:53:43.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691876 2024-08-19T14:53:43.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691876 2024-08-19T14:53:43.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673035 2-64424509508 3-300647710730 4-107374182452 5-347892350978 6-150323855403 7-171798691876' 2024-08-19T14:53:43.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:43.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T14:53:43.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528346 2024-08-19T14:53:43.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528346 2024-08-19T14:53:43.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673035 2-64424509508 3-300647710730 4-107374182452 5-347892350978 6-150323855403 7-171798691876 8-193273528346' 2024-08-19T14:53:43.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:53:43.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T14:53:43.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547206 2024-08-19T14:53:43.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547206 2024-08-19T14:53:43.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673035 2-64424509508 3-300647710730 4-107374182452 5-347892350978 6-150323855403 7-171798691876 8-193273528346 9-322122547206' 2024-08-19T14:53:43.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:43.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836565 2024-08-19T14:53:43.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:43.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:53:43.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836565 2024-08-19T14:53:43.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:43.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836565 2024-08-19T14:53:43.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836565' 2024-08-19T14:53:43.735 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836565 2024-08-19T14:53:43.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:53:44.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836564 -lt 21474836565 2024-08-19T14:53:44.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:53:45.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:53:45.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:53:45.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836566 -lt 21474836565 2024-08-19T14:53:45.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:45.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673035 2024-08-19T14:53:45.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:45.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:53:45.281 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673035 2024-08-19T14:53:45.281 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:45.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673035 2024-08-19T14:53:45.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673035' 2024-08-19T14:53:45.282 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949673035 2024-08-19T14:53:45.282 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:53:45.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673035 -lt 42949673035 2024-08-19T14:53:45.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:45.552 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509508 2024-08-19T14:53:45.552 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:45.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:53:45.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509508 2024-08-19T14:53:45.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:45.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509508 2024-08-19T14:53:45.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509508' 2024-08-19T14:53:45.555 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509508 2024-08-19T14:53:45.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:53:45.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509509 -lt 64424509508 2024-08-19T14:53:45.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:45.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710730 2024-08-19T14:53:45.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:45.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:53:45.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710730 2024-08-19T14:53:45.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:45.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710730 2024-08-19T14:53:45.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710730' 2024-08-19T14:53:45.825 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.3 seq 300647710730 2024-08-19T14:53:45.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:53:46.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710730 -lt 300647710730 2024-08-19T14:53:46.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:46.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182452 2024-08-19T14:53:46.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:46.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T14:53:46.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182452 2024-08-19T14:53:46.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:46.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182452 2024-08-19T14:53:46.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182452' 2024-08-19T14:53:46.095 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.4 seq 107374182452 2024-08-19T14:53:46.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T14:53:46.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182452 -lt 107374182452 2024-08-19T14:53:46.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:46.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-347892350978 2024-08-19T14:53:46.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:46.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T14:53:46.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-347892350978 2024-08-19T14:53:46.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:46.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350978 2024-08-19T14:53:46.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 347892350978' 2024-08-19T14:53:46.368 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.5 seq 347892350978 2024-08-19T14:53:46.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T14:53:46.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350979 -lt 347892350978 2024-08-19T14:53:46.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:46.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855403 2024-08-19T14:53:46.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:46.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T14:53:46.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855403 2024-08-19T14:53:46.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:46.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855403 2024-08-19T14:53:46.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855403' 2024-08-19T14:53:46.639 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.6 seq 150323855403 2024-08-19T14:53:46.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T14:53:46.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855403 -lt 150323855403 2024-08-19T14:53:46.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:46.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691876 2024-08-19T14:53:46.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:46.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T14:53:46.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691876 2024-08-19T14:53:46.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:46.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691876 2024-08-19T14:53:46.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691876' 2024-08-19T14:53:46.928 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.7 seq 171798691876 2024-08-19T14:53:46.929 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T14:53:47.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691877 -lt 171798691876 2024-08-19T14:53:47.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:47.202 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528346 2024-08-19T14:53:47.202 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:47.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T14:53:47.203 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528346 2024-08-19T14:53:47.203 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:47.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528346 2024-08-19T14:53:47.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528346' 2024-08-19T14:53:47.205 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.8 seq 193273528346 2024-08-19T14:53:47.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T14:53:47.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528346 -lt 193273528346 2024-08-19T14:53:47.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:53:47.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-322122547206 2024-08-19T14:53:47.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:53:47.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T14:53:47.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-322122547206 2024-08-19T14:53:47.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:53:47.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547206 2024-08-19T14:53:47.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 322122547206' 2024-08-19T14:53:47.477 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.9 seq 322122547206 2024-08-19T14:53:47.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T14:53:47.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547206 -lt 322122547206 2024-08-19T14:53:47.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:53:47.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:53:47.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:53:48.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:53:48.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:53:48.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:53:48.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:53:48.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:53:48.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:53:48.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:53:48.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:53:48.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:53:48.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:53:48.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:53:48.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:53:48.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:53:48.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:53:48.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:53:48.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-08-19T14:53:48.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T14:53:48.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T14:53:48.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:53:48.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:53:48.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-08-19T14:53:48.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-08-19T14:53:48.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-08-19T14:53:48.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-08-19T14:53:48.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:53:48.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:53:48.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:53:48.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:53:49.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T14:53:17.431048+0000 2024-08-19T14:53:49.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-08-19T14:53:49.514 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-08-19T14:53:49.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-08-19T14:53:17.431048+0000 2024-08-19T14:53:49.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T14:53:49.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:53:17.431048+0000 2024-08-19T14:53:49.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:53:49.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:53:49.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:53:49.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:53:49.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:53:49.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:53:49.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:53:49.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:53:49.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:53:17.431048+0000 '>' 2024-08-19T14:53:17.431048+0000 2024-08-19T14:53:49.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:53:50.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:53:50.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:53:50.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:53:50.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:53:50.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:53:50.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:53:50.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:53:51.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:53:17.431048+0000 '>' 2024-08-19T14:53:17.431048+0000 2024-08-19T14:53:51.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:53:52.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:53:52.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:53:52.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:53:52.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:53:52.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:53:52.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:53:52.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:53:52.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:53:17.431048+0000 '>' 2024-08-19T14:53:17.431048+0000 2024-08-19T14:53:52.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:53:53.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:53:53.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:53:53.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:53:53.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:53:53.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:53:53.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:53:53.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:53:53.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:53:17.431048+0000 '>' 2024-08-19T14:53:17.431048+0000 2024-08-19T14:53:53.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:53:54.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:53:54.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:53:54.622 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:53:54.622 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:53:54.622 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:53:54.622 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:53:54.623 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:53:54.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:53:17.431048+0000 '>' 2024-08-19T14:53:17.431048+0000 2024-08-19T14:53:54.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:53:55.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:53:55.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:53:55.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:53:55.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:53:55.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:53:55.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:53:55.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:53:56.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:53:17.431048+0000 '>' 2024-08-19T14:53:17.431048+0000 2024-08-19T14:53:56.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:53:57.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:53:57.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:53:57.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:53:57.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:53:57.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:53:57.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:53:57.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:53:57.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:53:50.476942+0000 '>' 2024-08-19T14:53:17.431048+0000 2024-08-19T14:53:57.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:53:57.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T14:53:57.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:53:57.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:53:57.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-08-19T14:53:57.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:53:57.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T14:53:57.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:53:57.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:53:57.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-08-19T14:53:57.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:53:57.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-08-19T14:53:57.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:53:57.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:53:57.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:53:57.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:53:57.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:53:57.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:53:57.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T14:53:57.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:53:57.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:53:57.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-08-19T14:53:57.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:53:57.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-08-19T14:53:57.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-08-19T14:53:59.877 INFO:tasks.workunit.client.0.smithi102.stdout:_ 2024-08-19T14:53:59.877 INFO:tasks.workunit.client.0.smithi102.stdout:hinfo_key 2024-08-19T14:53:59.877 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 2024-08-19T14:54:00.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-08-19T14:54:00.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:54:00.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:54:00.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-08-19T14:54:00.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:54:00.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-08-19T14:54:00.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:54:00.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:54:00.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:54:00.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:54:00.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-08-19T14:54:00.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-08-19T14:54:00.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:54:00.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:54:00.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:54:00.151 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:54:00.151 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:54:00.151 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:54:00.151 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:54:00.151 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:54:00.151 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:54:00.151 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:54:00.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:54:00.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:54:00.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:54:00.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:54:00.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:54:00.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:54:00.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:54:00.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:54:00.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:54:00.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:54:00.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-08-19T14:54:00.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-08-19T14:54:00.152 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.9 2024-08-19T14:54:00.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:54:00.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-08-19T14:54:00.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-08-19T14:54:00.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:54:00.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:54:00.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:54:00.181 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:54:00.180+0000 7fcaf4db1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:00.183 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:54:00.182+0000 7fcaf4db1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:00.184 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:54:00.184+0000 7fcaf4db1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:00.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-08-19T14:54:00.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:54:00.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-08-19T14:54:00.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:54:00.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:54:00.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:00.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:54:00.438 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T14:54:00.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:00.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:54:00.700 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:54:00.700+0000 7fcaf4db1640 -1 Falling back to public interface 2024-08-19T14:54:00.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:01.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:01.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:01.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:54:01.718 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:54:01.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:01.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:54:01.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:02.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:02.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:02.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:54:02.992 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:54:02.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:02.993 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:54:02.990+0000 7fcaf4db1640 -1 osd.9 84 log_to_monitors true 2024-08-19T14:54:02.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:54:03.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:04.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:04.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:04.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:54:04.342 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:54:04.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:04.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:54:04.936 INFO:tasks.workunit.client.0.smithi102.stderr:osd.9 up in weight 1 up_from 88 up_thru 0 down_at 84 last_clean_interval [75,84) [v2:127.0.0.1:6874/1242758444,v1:127.0.0.1:6875/1242758444] [v2:127.0.0.1:6876/1242758444,v1:127.0.0.1:6877/1242758444] exists,up d4d5322d-f118-4a96-ac94-665aa58c0de8 2024-08-19T14:54:04.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:54:04.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:54:04.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:54:04.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:54:04.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:54:04.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:54:04.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:54:04.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:54:04.937 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:54:04.937 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:54:04.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:54:04.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:54:04.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:54:05.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:54:05.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:54:05.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:54:05.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:54:05.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:54:05.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:54:05.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:54:05.333 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T14:54:05.333 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T14:54:05.333 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T14:54:05.333 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-08-19T14:54:05.333 INFO:tasks.workunit.client.0.smithi102.stderr:5 2024-08-19T14:54:05.333 INFO:tasks.workunit.client.0.smithi102.stderr:6 2024-08-19T14:54:05.333 INFO:tasks.workunit.client.0.smithi102.stderr:7 2024-08-19T14:54:05.333 INFO:tasks.workunit.client.0.smithi102.stderr:8 2024-08-19T14:54:05.333 INFO:tasks.workunit.client.0.smithi102.stderr:9' 2024-08-19T14:54:05.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:54:05.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:05.334 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:54:05.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836571 2024-08-19T14:54:05.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836571 2024-08-19T14:54:05.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571' 2024-08-19T14:54:05.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:05.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:54:05.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673040 2024-08-19T14:54:05.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673040 2024-08-19T14:54:05.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673040' 2024-08-19T14:54:05.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:05.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:54:05.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509514 2024-08-19T14:54:05.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509514 2024-08-19T14:54:05.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673040 2-64424509514' 2024-08-19T14:54:05.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:05.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:54:05.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710736 2024-08-19T14:54:05.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710736 2024-08-19T14:54:05.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673040 2-64424509514 3-300647710736' 2024-08-19T14:54:05.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:05.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T14:54:05.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182458 2024-08-19T14:54:05.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182458 2024-08-19T14:54:05.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673040 2-64424509514 3-300647710736 4-107374182458' 2024-08-19T14:54:05.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:05.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T14:54:05.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350984 2024-08-19T14:54:05.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350984 2024-08-19T14:54:05.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673040 2-64424509514 3-300647710736 4-107374182458 5-347892350984' 2024-08-19T14:54:05.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:05.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T14:54:06.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855408 2024-08-19T14:54:06.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855408 2024-08-19T14:54:06.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673040 2-64424509514 3-300647710736 4-107374182458 5-347892350984 6-150323855408' 2024-08-19T14:54:06.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:06.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T14:54:06.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691882 2024-08-19T14:54:06.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691882 2024-08-19T14:54:06.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673040 2-64424509514 3-300647710736 4-107374182458 5-347892350984 6-150323855408 7-171798691882' 2024-08-19T14:54:06.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:06.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T14:54:06.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528351 2024-08-19T14:54:06.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528351 2024-08-19T14:54:06.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673040 2-64424509514 3-300647710736 4-107374182458 5-347892350984 6-150323855408 7-171798691882 8-193273528351' 2024-08-19T14:54:06.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:06.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T14:54:06.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-08-19T14:54:06.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-08-19T14:54:06.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673040 2-64424509514 3-300647710736 4-107374182458 5-347892350984 6-150323855408 7-171798691882 8-193273528351 9-377957122050' 2024-08-19T14:54:06.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:06.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836571 2024-08-19T14:54:06.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:06.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:54:06.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836571 2024-08-19T14:54:06.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:06.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836571 2024-08-19T14:54:06.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836571' 2024-08-19T14:54:06.382 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836571 2024-08-19T14:54:06.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:54:06.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836571 -lt 21474836571 2024-08-19T14:54:06.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:06.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673040 2024-08-19T14:54:06.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:06.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:54:06.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673040 2024-08-19T14:54:06.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:06.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673040 2024-08-19T14:54:06.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673040' 2024-08-19T14:54:06.661 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949673040 2024-08-19T14:54:06.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:54:06.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673041 -lt 42949673040 2024-08-19T14:54:06.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:06.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509514 2024-08-19T14:54:06.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:06.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:54:06.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509514 2024-08-19T14:54:06.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:06.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509514 2024-08-19T14:54:06.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509514' 2024-08-19T14:54:06.933 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509514 2024-08-19T14:54:06.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:54:07.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509514 -lt 64424509514 2024-08-19T14:54:07.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:07.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710736 2024-08-19T14:54:07.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:07.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:54:07.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710736 2024-08-19T14:54:07.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:07.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710736 2024-08-19T14:54:07.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710736' 2024-08-19T14:54:07.217 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.3 seq 300647710736 2024-08-19T14:54:07.217 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:54:07.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710736 -lt 300647710736 2024-08-19T14:54:07.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:07.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182458 2024-08-19T14:54:07.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:07.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T14:54:07.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182458 2024-08-19T14:54:07.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:07.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182458 2024-08-19T14:54:07.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182458' 2024-08-19T14:54:07.488 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.4 seq 107374182458 2024-08-19T14:54:07.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T14:54:07.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182458 -lt 107374182458 2024-08-19T14:54:07.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:07.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-347892350984 2024-08-19T14:54:07.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:07.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T14:54:07.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-347892350984 2024-08-19T14:54:07.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:07.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350984 2024-08-19T14:54:07.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 347892350984' 2024-08-19T14:54:07.760 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.5 seq 347892350984 2024-08-19T14:54:07.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T14:54:08.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350984 -lt 347892350984 2024-08-19T14:54:08.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:08.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855408 2024-08-19T14:54:08.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:08.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T14:54:08.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855408 2024-08-19T14:54:08.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:08.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855408 2024-08-19T14:54:08.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855408' 2024-08-19T14:54:08.031 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.6 seq 150323855408 2024-08-19T14:54:08.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T14:54:08.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855408 -lt 150323855408 2024-08-19T14:54:08.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:08.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691882 2024-08-19T14:54:08.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:08.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T14:54:08.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691882 2024-08-19T14:54:08.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:08.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691882 2024-08-19T14:54:08.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691882' 2024-08-19T14:54:08.308 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.7 seq 171798691882 2024-08-19T14:54:08.308 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T14:54:08.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691882 -lt 171798691882 2024-08-19T14:54:08.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:08.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528351 2024-08-19T14:54:08.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:08.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T14:54:08.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528351 2024-08-19T14:54:08.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:08.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528351 2024-08-19T14:54:08.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528351' 2024-08-19T14:54:08.581 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.8 seq 193273528351 2024-08-19T14:54:08.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T14:54:08.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528352 -lt 193273528351 2024-08-19T14:54:08.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:08.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-377957122050 2024-08-19T14:54:08.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:08.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T14:54:08.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-377957122050 2024-08-19T14:54:08.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:08.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-08-19T14:54:08.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 377957122050' 2024-08-19T14:54:08.853 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.9 seq 377957122050 2024-08-19T14:54:08.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T14:54:09.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122051 -lt 377957122050 2024-08-19T14:54:09.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:54:09.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:09.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:09.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:54:09.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:54:09.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:54:09.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:54:09.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:54:09.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:54:09.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:54:09.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:54:09.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:54:09.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:54:09.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:09.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:10.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:54:10.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:54:10.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:54:10.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T14:54:10.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T14:54:10.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-08-19T14:54:10.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-08-19T14:54:10.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-08-19T14:54:10.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-08-19T14:54:10.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-08-19T14:54:10.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-08-19T14:54:10.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T14:54:10.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:54:10.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:54:10.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 216698"' 2024-08-19T14:54:10.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 216698' 2024-08-19T14:54:10.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-08-19T14:54:10.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:54:10.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:54:10.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 216700"' 2024-08-19T14:54:10.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T14:54:10.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 216700' 2024-08-19T14:54:10.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-08-19T14:54:10.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 216698 216700' 2024-08-19T14:54:10.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-08-19T14:54:10.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:54:10.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 216698 2024-08-19T14:54:10.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/216701: /' 2024-08-19T14:54:10.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-08-19T14:54:10.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/216703: /' 2024-08-19T14:54:13.255 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:54:13.255 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:54:13.255 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-08-19T14:54:13.255 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:54:13.256 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-08-19T14:54:13.256 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:54:13.256 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:54:13.256 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-08-19T14:54:13.256 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:54:13.256 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-08-19T14:54:13.256 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:54:13.256 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:54:13.256 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:54:13.256 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:54:13.256 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:54:13.256 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:54:13.257 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-08-19T14:54:13.257 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:54:13.257 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:54:13.257 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-08-19T14:54:13.257 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:54:13.257 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-08-19T14:54:13.257 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-08-19T14:54:13.257 INFO:tasks.workunit.client.0.smithi102.stderr:216703: remove 2#2:eb822e21:::SOMETHING:head# 2024-08-19T14:54:13.257 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-08-19T14:54:13.257 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:54:13.257 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:54:13.257 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-08-19T14:54:13.258 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:54:13.258 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-08-19T14:54:13.258 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:54:13.258 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:54:13.258 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:54:13.259 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:54:13.259 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-08-19T14:54:13.259 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-08-19T14:54:13.259 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:54:13.259 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:54:13.259 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:54:13.259 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:54:13.259 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:54:13.259 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:54:13.259 INFO:tasks.workunit.client.0.smithi102.stderr:216703: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:54:13.259 INFO:tasks.workunit.client.0.smithi102.stderr:216703: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:54:13.259 INFO:tasks.workunit.client.0.smithi102.stderr:216703: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:54:13.260 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:54:13.260 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:54:13.260 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:54:13.260 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:54:13.260 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:54:13.260 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:54:13.260 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:54:13.260 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:54:13.260 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:54:13.260 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:54:13.260 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:54:13.261 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-08-19T14:54:13.261 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-08-19T14:54:13.261 INFO:tasks.workunit.client.0.smithi102.stderr:216703: start osd.9 2024-08-19T14:54:13.261 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:54:13.274 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:54:13.274 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:54:13.274 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-08-19T14:54:13.275 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:54:13.275 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T14:54:13.275 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:54:13.275 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:54:13.275 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-08-19T14:54:13.275 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:54:13.275 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-08-19T14:54:13.275 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:54:13.275 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:54:13.275 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:54:13.275 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:54:13.275 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:54:13.275 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:54:13.275 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T14:54:13.276 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:54:13.276 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:54:13.276 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-08-19T14:54:13.276 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:54:13.276 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-08-19T14:54:13.276 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-08-19T14:54:13.276 INFO:tasks.workunit.client.0.smithi102.stderr:216701: remove 1#2:eb822e21:::SOMETHING:head# 2024-08-19T14:54:13.276 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-08-19T14:54:13.276 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:54:13.276 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:54:13.276 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-08-19T14:54:13.276 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:54:13.276 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-08-19T14:54:13.277 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:54:13.277 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:54:13.278 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:54:13.278 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:54:13.278 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-08-19T14:54:13.278 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-08-19T14:54:13.278 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:54:13.279 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:54:13.279 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:54:13.279 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:54:13.279 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:54:13.279 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:54:13.279 INFO:tasks.workunit.client.0.smithi102.stderr:216701: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:54:13.279 INFO:tasks.workunit.client.0.smithi102.stderr:216701: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:54:13.279 INFO:tasks.workunit.client.0.smithi102.stderr:216701: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:54:13.279 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:54:13.279 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:54:13.279 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:54:13.279 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:54:13.280 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:54:13.280 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:54:13.280 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:54:13.280 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:54:13.280 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:54:13.280 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:54:13.280 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:54:13.280 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-08-19T14:54:13.280 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-08-19T14:54:13.280 INFO:tasks.workunit.client.0.smithi102.stderr:216701: start osd.5 2024-08-19T14:54:13.280 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:54:17.830 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-08-19T14:54:17.831 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-08-19T14:54:17.831 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:54:17.831 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:54:17.831 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:54:17.831 INFO:tasks.workunit.client.0.smithi102.stderr:216703: 2024-08-19T14:54:13.291+0000 7f3c2cc16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:17.831 INFO:tasks.workunit.client.0.smithi102.stderr:216703: 2024-08-19T14:54:13.294+0000 7f3c2cc16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:17.831 INFO:tasks.workunit.client.0.smithi102.stderr:216703: 2024-08-19T14:54:13.295+0000 7f3c2cc16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:17.831 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-08-19T14:54:17.831 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:54:17.831 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-08-19T14:54:17.831 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:54:17.832 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:54:17.832 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:17.832 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:54:17.832 INFO:tasks.workunit.client.0.smithi102.stderr:216703: 0 2024-08-19T14:54:17.832 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:17.832 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:54:17.832 INFO:tasks.workunit.client.0.smithi102.stderr:216703: 2024-08-19T14:54:13.823+0000 7f3c2cc16640 -1 Falling back to public interface 2024-08-19T14:54:17.832 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:17.832 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:17.832 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:17.832 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:54:17.832 INFO:tasks.workunit.client.0.smithi102.stderr:216703: 1 2024-08-19T14:54:17.832 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:17.833 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:54:17.833 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:17.833 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:17.833 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:17.833 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:54:17.833 INFO:tasks.workunit.client.0.smithi102.stderr:216703: 2 2024-08-19T14:54:17.833 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:17.833 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:54:17.833 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:17.833 INFO:tasks.workunit.client.0.smithi102.stderr:216703: 2024-08-19T14:54:16.441+0000 7f3c2cc16640 -1 osd.9 90 log_to_monitors true 2024-08-19T14:54:17.833 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:17.833 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:17.833 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:54:17.834 INFO:tasks.workunit.client.0.smithi102.stderr:216703: 3 2024-08-19T14:54:17.834 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:17.834 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:54:18.091 INFO:tasks.workunit.client.0.smithi102.stderr:216703: osd.9 up in weight 1 up_from 96 up_thru 0 down_at 91 last_c216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-08-19T14:54:18.091 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-08-19T14:54:18.091 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:54:18.091 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:54:18.091 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:54:18.091 INFO:tasks.workunit.client.0.smithi102.stderr:216701: 2024-08-19T14:54:13.307+0000 7ff4c360e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:18.092 INFO:tasks.workunit.client.0.smithi102.stderr:216701: 2024-08-19T14:54:13.309+0000 7ff4c360e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:18.092 INFO:tasks.workunit.client.0.smithi102.stderr:216701: 2024-08-19T14:54:13.311+0000 7ff4c360e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:18.092 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-08-19T14:54:18.092 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:54:18.092 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-08-19T14:54:18.092 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:54:18.092 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:54:18.092 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:18.092 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:54:18.092 INFO:tasks.workunit.client.0.smithi102.stderr:216701: 0 2024-08-19T14:54:18.092 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:18.092 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:54:18.093 INFO:tasks.workunit.client.0.smithi102.stderr:216701: 2024-08-19T14:54:13.831+0000 7ff4c360e640 -1 Falling back to public interface 2024-08-19T14:54:18.093 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:18.093 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:18.093 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:18.093 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:54:18.093 INFO:tasks.workunit.client.0.smithi102.stderr:216701: 1 2024-08-19T14:54:18.093 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:18.093 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:54:18.093 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:18.093 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:18.093 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:18.093 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:54:18.093 INFO:tasks.workunit.client.0.smithi102.stderr:216701: 2 2024-08-19T14:54:18.093 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:18.094 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:54:18.094 INFO:tasks.workunit.client.0.smithi102.stderr:216701: 2024-08-19T14:54:16.490+0000 7ff4c360e640 -1 osd.5 90 log_to_monitors true 2024-08-19T14:54:18.094 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:18.094 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:18.094 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:18.094 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:54:18.094 INFO:tasks.workunit.client.0.smithi102.stderr:216701: 3 2024-08-19T14:54:18.094 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:18.094 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:54:18.927 INFO:tasks.workunit.client.0.smithi102.stderr:216701: osd.5 up in weight 1 up_from 97 up_thru 81 down_at 91 last_lean_interval [88,90) [v2:127.0.0.1:6842/3721068793,v1:127.0.0.1:6843/3721068793] [v2:127.0.0.1:6844/3721068793,v1:127.0.0.1:6845/3721068793] exists,up d4d5322d-f118-4a96-ac94-665aa58c0de8 2024-08-19T14:54:18.927 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:54:18.927 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:54:18.927 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:54:18.927 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:54:18.927 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:54:18.927 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:54:18.928 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:54:18.928 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:54:18.928 INFO:tasks.workunit.client.0.smithi102.stderr:216703: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:54:18.928 INFO:tasks.workunit.client.0.smithi102.stderr:216703: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:54:18.928 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:54:18.928 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:54:18.928 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:54:18.928 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:54:18.928 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:54:18.928 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:54:18.928 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:54:18.929 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:54:18.929 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:54:18.929 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:54:18.929 INFO:tasks.workunit.client.0.smithi102.stderr:216703: 1 2024-08-19T14:54:18.929 INFO:tasks.workunit.client.0.smithi102.stderr:216703: 2 2024-08-19T14:54:18.929 INFO:tasks.workunit.client.0.smithi102.stderr:216703: 3 2024-08-19T14:54:18.929 INFO:tasks.workunit.client.0.smithi102.stderr:216703: 4 2024-08-19T14:54:18.929 INFO:tasks.workunit.client.0.smithi102.stderr:216703: 5 2024-08-19T14:54:18.929 INFO:tasks.workunit.client.0.smithi102.stderr:216703: 6 2024-08-19T14:54:18.929 INFO:tasks.workunit.client.0.smithi102.stderr:216703: 7 2024-08-19T14:54:18.930 INFO:tasks.workunit.client.0.smithi102.stderr:216703: 8 2024-08-19T14:54:18.930 INFO:tasks.workunit.client.0.smithi102.stderr:216703: 9' 2024-08-19T14:54:18.930 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:54:18.930 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:18.930 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:54:18.930 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836574 2024-08-19T14:54:18.930 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836574 2024-08-19T14:54:18.930 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574' 2024-08-19T14:54:18.930 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:18.930 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:54:18.930 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673044 2024-08-19T14:54:18.931 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673044 2024-08-19T14:54:18.931 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673044' 2024-08-19T14:54:18.931 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:19.171 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qaclean_interval [81,90) [v2:127.0.0.1:6874/3030993767,v1:127.0.0.1:6875/3030993767] [v2:127.0.0.1:6876/3030993767,v1:127.0.0.1:6877/3030993767] exists,up 7c59bef0-d717-44f2-b3df-88b33605d1af 2024-08-19T14:54:19.171 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:54:19.171 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:54:19.171 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:54:19.171 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:54:19.171 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:54:19.171 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:54:19.171 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:54:19.171 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:54:19.171 INFO:tasks.workunit.client.0.smithi102.stderr:216701: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:54:19.172 INFO:tasks.workunit.client.0.smithi102.stderr:216701: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:54:19.172 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:54:19.172 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:54:19.172 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:54:19.172 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:54:19.172 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:54:19.172 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:54:19.172 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:54:19.172 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:54:19.172 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:54:19.172 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:54:19.172 INFO:tasks.workunit.client.0.smithi102.stderr:216701: 1 2024-08-19T14:54:19.172 INFO:tasks.workunit.client.0.smithi102.stderr:216701: 2 2024-08-19T14:54:19.173 INFO:tasks.workunit.client.0.smithi102.stderr:216701: 3 2024-08-19T14:54:19.173 INFO:tasks.workunit.client.0.smithi102.stderr:216701: 4 2024-08-19T14:54:19.173 INFO:tasks.workunit.client.0.smithi102.stderr:216701: 5 2024-08-19T14:54:19.173 INFO:tasks.workunit.client.0.smithi102.stderr:216701: 6 2024-08-19T14:54:19.173 INFO:tasks.workunit.client.0.smithi102.stderr:216701: 7 2024-08-19T14:54:19.173 INFO:tasks.workunit.client.0.smithi102.stderr:216701: 8 2024-08-19T14:54:19.173 INFO:tasks.workunit.client.0.smithi102.stderr:216701: 9' 2024-08-19T14:54:19.173 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:54:19.173 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:19.173 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:54:19.173 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836575 2024-08-19T14:54:19.173 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836575 2024-08-19T14:54:19.173 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575' 2024-08-19T14:54:19.173 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:19.174 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:54:19.174 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673045 2024-08-19T14:54:19.174 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673045 2024-08-19T14:54:19.174 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673045' 2024-08-19T14:54:19.174 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:20.240 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/q/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:54:20.240 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509518 2024-08-19T14:54:20.240 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509518 2024-08-19T14:54:20.240 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673044 2-64424509518' 2024-08-19T14:54:20.241 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:20.241 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:54:20.241 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710739 2024-08-19T14:54:20.241 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710739 2024-08-19T14:54:20.241 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673044 2-64424509518 3-300647710739' 2024-08-19T14:54:20.241 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:20.241 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T14:54:20.241 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182461 2024-08-19T14:54:20.241 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182461 2024-08-19T14:54:20.241 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673044 2-64424509518 3-300647710739 4-107374182461' 2024-08-19T14:54:20.241 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:20.241 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T14:54:20.241 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827714 2024-08-19T14:54:20.242 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827714 2024-08-19T14:54:20.242 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673044 2-64424509518 3-300647710739 4-107374182461 5-416611827714' 2024-08-19T14:54:20.242 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:20.242 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T14:54:20.242 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855412 2024-08-19T14:54:20.242 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855412 2024-08-19T14:54:20.242 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673044 2-64424509518 3-300647710739 4-107374182461 5-416611827714 6-150323855412' 2024-08-19T14:54:20.242 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:20.242 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T14:54:20.242 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691886 2024-08-19T14:54:20.242 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691886 2024-08-19T14:54:20.242 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673044 2-64424509518 3-300647710739 4-107374182461 5-416611827714 6-150323855412 7-171798691886' 2024-08-19T14:54:20.243 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:20.243 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T14:54:20.347 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clonea/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:54:20.347 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509519 2024-08-19T14:54:20.347 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509519 2024-08-19T14:54:20.347 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673045 2-64424509519' 2024-08-19T14:54:20.347 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:20.347 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:54:20.347 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710740 2024-08-19T14:54:20.347 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710740 2024-08-19T14:54:20.347 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673045 2-64424509519 3-300647710740' 2024-08-19T14:54:20.347 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:20.348 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T14:54:20.348 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182462 2024-08-19T14:54:20.348 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182462 2024-08-19T14:54:20.348 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673045 2-64424509519 3-300647710740 4-107374182462' 2024-08-19T14:54:20.348 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:20.348 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T14:54:20.348 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827715 2024-08-19T14:54:20.348 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827715 2024-08-19T14:54:20.348 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673045 2-64424509519 3-300647710740 4-107374182462 5-416611827715' 2024-08-19T14:54:20.348 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:20.348 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T14:54:20.348 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855413 2024-08-19T14:54:20.348 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855413 2024-08-19T14:54:20.349 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673045 2-64424509519 3-300647710740 4-107374182462 5-416611827715 6-150323855413' 2024-08-19T14:54:20.349 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:20.349 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T14:54:20.349 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691887 2024-08-19T14:54:20.349 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691887 2024-08-19T14:54:20.349 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673045 2-64424509519 3-300647710740 4-107374182462 5-416611827715 6-150323855413 7-171798691887' 2024-08-19T14:54:20.349 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:20.349 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T14:54:20.974 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clon.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528355 2024-08-19T14:54:20.975 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528355 2024-08-19T14:54:20.975 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673044 2-64424509518 3-300647710739 4-107374182461 5-416611827714 6-150323855412 7-171798691886 8-193273528355' 2024-08-19T14:54:20.975 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:20.975 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T14:54:20.975 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860418 2024-08-19T14:54:20.975 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860418 2024-08-19T14:54:20.975 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673044 2-64424509518 3-300647710739 4-107374182461 5-416611827714 6-150323855412 7-171798691886 8-193273528355 9-412316860418' 2024-08-19T14:54:20.975 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:20.975 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836574 2024-08-19T14:54:20.975 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:20.976 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:54:20.976 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836574 2024-08-19T14:54:20.976 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:20.976 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836574 2024-08-19T14:54:20.976 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836574' 2024-08-19T14:54:20.976 INFO:tasks.workunit.client.0.smithi102.stderr:216703: waiting osd.0 seq 21474836574 2024-08-19T14:54:20.976 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:54:20.976 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836576 -lt 21474836574 2024-08-19T14:54:20.976 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:20.976 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673044 2024-08-19T14:54:20.976 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:20.976 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:54:20.976 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673044 2024-08-19T14:54:20.977 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:20.977 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673044 2024-08-19T14:54:20.977 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673044' 2024-08-19T14:54:20.977 INFO:tasks.workunit.client.0.smithi102.stderr:216703: waiting osd.1 seq 42949673044 2024-08-19T14:54:20.977 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:54:20.977 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673045 -lt 42949673044 2024-08-19T14:54:20.977 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:20.977 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509518 2024-08-19T14:54:20.977 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:20.977 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:54:21.057 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.ce.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528356 2024-08-19T14:54:21.057 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528356 2024-08-19T14:54:21.057 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673045 2-64424509519 3-300647710740 4-107374182462 5-416611827715 6-150323855413 7-171798691887 8-193273528356' 2024-08-19T14:54:21.057 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:21.058 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T14:54:21.058 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860419 2024-08-19T14:54:21.058 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860419 2024-08-19T14:54:21.058 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673045 2-64424509519 3-300647710740 4-107374182462 5-416611827715 6-150323855413 7-171798691887 8-193273528356 9-412316860419' 2024-08-19T14:54:21.058 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:21.058 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836575 2024-08-19T14:54:21.058 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:21.058 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:54:21.058 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836575 2024-08-19T14:54:21.058 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:21.058 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836575 2024-08-19T14:54:21.058 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836575' 2024-08-19T14:54:21.059 INFO:tasks.workunit.client.0.smithi102.stderr:216701: waiting osd.0 seq 21474836575 2024-08-19T14:54:21.059 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:54:21.059 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836576 -lt 21474836575 2024-08-19T14:54:21.059 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:21.059 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673045 2024-08-19T14:54:21.059 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:21.059 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:54:21.059 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673045 2024-08-19T14:54:21.059 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:21.059 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673045 2024-08-19T14:54:21.059 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673045' 2024-08-19T14:54:21.059 INFO:tasks.workunit.client.0.smithi102.stderr:216701: waiting osd.1 seq 42949673045 2024-08-19T14:54:21.059 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:54:21.060 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673045 -lt 42949673045 2024-08-19T14:54:21.060 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:21.060 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509519 2024-08-19T14:54:21.060 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:21.060 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:54:21.919 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.lient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509518 2024-08-19T14:54:21.920 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:21.920 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509518 2024-08-19T14:54:21.920 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509518' 2024-08-19T14:54:21.920 INFO:tasks.workunit.client.0.smithi102.stderr:216703: waiting osd.2 seq 64424509518 2024-08-19T14:54:21.920 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:54:21.920 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509519 -lt 64424509518 2024-08-19T14:54:21.920 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:21.920 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710739 2024-08-19T14:54:21.920 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:21.920 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:54:21.920 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710739 2024-08-19T14:54:21.920 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:21.920 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710739 2024-08-19T14:54:21.921 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710739' 2024-08-19T14:54:21.921 INFO:tasks.workunit.client.0.smithi102.stderr:216703: waiting osd.3 seq 300647710739 2024-08-19T14:54:21.921 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:54:21.921 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710741 -lt 300647710739 2024-08-19T14:54:21.921 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:21.921 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182461 2024-08-19T14:54:21.921 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:21.921 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T14:54:21.921 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182461 2024-08-19T14:54:21.921 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:21.921 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182461 2024-08-19T14:54:21.921 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182461' 2024-08-19T14:54:21.921 INFO:tasks.workunit.client.0.smithi102.stderr:216703: waiting osd.4 seq 107374182461 2024-08-19T14:54:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T14:54:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182463 -lt 107374182461 2024-08-19T14:54:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-416611827714 2024-08-19T14:54:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T14:54:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-416611827714 2024-08-19T14:54:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827714 2024-08-19T14:54:21.962 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiticlient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509519 2024-08-19T14:54:21.962 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:21.962 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509519 2024-08-19T14:54:21.962 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509519' 2024-08-19T14:54:21.963 INFO:tasks.workunit.client.0.smithi102.stderr:216701: waiting osd.2 seq 64424509519 2024-08-19T14:54:21.963 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:54:21.963 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509519 -lt 64424509519 2024-08-19T14:54:21.963 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:21.963 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710740 2024-08-19T14:54:21.963 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:21.963 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:54:21.963 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710740 2024-08-19T14:54:21.963 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:21.963 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710740 2024-08-19T14:54:21.963 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710740' 2024-08-19T14:54:21.963 INFO:tasks.workunit.client.0.smithi102.stderr:216701: waiting osd.3 seq 300647710740 2024-08-19T14:54:21.964 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:54:21.964 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710741 -lt 300647710740 2024-08-19T14:54:21.964 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:21.964 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182462 2024-08-19T14:54:21.964 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:21.964 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T14:54:21.964 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182462 2024-08-19T14:54:21.964 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:21.964 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182462 2024-08-19T14:54:21.964 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182462' 2024-08-19T14:54:21.964 INFO:tasks.workunit.client.0.smithi102.stderr:216701: waiting osd.4 seq 107374182462 2024-08-19T14:54:21.964 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T14:54:21.964 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182463 -lt 107374182462 2024-08-19T14:54:21.965 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:21.965 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-416611827715 2024-08-19T14:54:21.965 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:21.965 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T14:54:21.965 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-416611827715 2024-08-19T14:54:21.965 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:21.965 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827715 2024-08-19T14:54:23.157 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 416611827715' 2024-08-19T14:54:23.157 INFO:tasks.workunit.client.0.smithi102.stderr:216701: waiting osd.5 seq 416611827715 2024-08-19T14:54:23.157 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T14:54:23.157 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827715 -lt 416611827715 2024-08-19T14:54:23.157 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:23.157 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855413 2024-08-19T14:54:23.157 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:23.157 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T14:54:23.157 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855413 2024-08-19T14:54:23.157 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:23.158 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855413 2024-08-19T14:54:23.158 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855413' 2024-08-19T14:54:23.158 INFO:tasks.workunit.client.0.smithi102.stderr:216701: waiting osd.6 seq 150323855413 2024-08-19T14:54:23.158 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T14:54:23.158 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855413 -lt 150323855413 2024-08-19T14:54:23.158 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:23.158 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691887 2024-08-19T14:54:23.158 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:23.158 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T14:54:23.158 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691887 2024-08-19T14:54:23.158 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:23.158 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691887 2024-08-19T14:54:23.158 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691887' 2024-08-19T14:54:23.159 INFO:tasks.workunit.client.0.smithi102.stderr:216701: waiting osd.7 seq 171798691887 2024-08-19T14:54:23.159 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T14:54:23.159 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691887 -lt 171798691887 2024-08-19T14:54:23.159 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:23.159 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528356 2024-08-19T14:54:23.159 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:23.159 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T14:54:23.159 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528356 2024-08-19T14:54:23.159 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:23.159 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528356 2024-08-19T14:54:23.159 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528356' 2024-08-19T14:54:23.159 INFO:tasks.workunit.client.0.smithi102.stderr:216701: waiting osd.8 seq 193273528356 2024-08-19T14:54:23.159 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T14:54:23.159 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528356 -lt 193273528356 2024-08-19T14:54:23.210 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_png osd.5 seq 416611827714' 2024-08-19T14:54:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:216703: waiting osd.5 seq 416611827714 2024-08-19T14:54:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T14:54:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827715 -lt 416611827714 2024-08-19T14:54:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855412 2024-08-19T14:54:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T14:54:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855412 2024-08-19T14:54:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855412 2024-08-19T14:54:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855412' 2024-08-19T14:54:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:216703: waiting osd.6 seq 150323855412 2024-08-19T14:54:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T14:54:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855413 -lt 150323855412 2024-08-19T14:54:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691886 2024-08-19T14:54:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T14:54:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691886 2024-08-19T14:54:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691886 2024-08-19T14:54:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691886' 2024-08-19T14:54:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:216703: waiting osd.7 seq 171798691886 2024-08-19T14:54:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T14:54:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691887 -lt 171798691886 2024-08-19T14:54:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528355 2024-08-19T14:54:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T14:54:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528355 2024-08-19T14:54:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528355 2024-08-19T14:54:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528355' 2024-08-19T14:54:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:216703: waiting osd.8 seq 193273528355 2024-08-19T14:54:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T14:54:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528356 -lt 193273528355 2024-08-19T14:54:24.529 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pgg_stats: for s in $seqs 2024-08-19T14:54:24.529 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-412316860419 2024-08-19T14:54:24.530 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:24.530 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T14:54:24.530 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-412316860419 2024-08-19T14:54:24.530 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:24.530 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860419 2024-08-19T14:54:24.530 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 412316860419' 2024-08-19T14:54:24.530 INFO:tasks.workunit.client.0.smithi102.stderr:216701: waiting osd.9 seq 412316860419 2024-08-19T14:54:24.530 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T14:54:24.530 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860420 -lt 412316860419 2024-08-19T14:54:24.530 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:54:24.530 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:24.530 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:24.531 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:54:24.531 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:54:24.531 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:54:24.531 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:54:24.531 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:54:24.531 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:54:24.531 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:54:24.531 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:54:24.531 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-08-19T14:54:24.531 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:54:24.531 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:24.531 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:24.532 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-08-19T14:54:24.532 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-08-19T14:54:24.532 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:54:24.532 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-08-19T14:54:24.532 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:54:24.532 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:54:24.532 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:54:24.532 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:54:24.532 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:54:24.665 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ub_stats: for s in $seqs 2024-08-19T14:54:24.665 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-412316860418 2024-08-19T14:54:24.665 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:24.665 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T14:54:24.665 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-412316860418 2024-08-19T14:54:24.665 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:24.665 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860418 2024-08-19T14:54:24.665 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 412316860418' 2024-08-19T14:54:24.665 INFO:tasks.workunit.client.0.smithi102.stderr:216703: waiting osd.9 seq 412316860418 2024-08-19T14:54:24.665 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T14:54:24.665 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860420 -lt 412316860418 2024-08-19T14:54:24.666 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:54:24.666 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:24.666 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:24.666 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:54:24.666 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:54:24.666 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:54:24.666 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:54:24.666 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:54:24.666 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:54:24.666 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:54:24.666 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:54:24.666 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-08-19T14:54:24.666 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:54:24.667 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:24.667 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:24.667 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-08-19T14:54:24.667 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-08-19T14:54:24.667 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:54:24.667 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-08-19T14:54:24.667 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:54:24.667 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:54:24.667 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:54:24.667 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:54:24.667 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:54:25.905 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:54:25.906 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:54:25.906 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:54:25.906 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:54:25.906 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:54:25.906 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:54:25.906 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:54:25.906 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:25.906 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:25.906 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-08-19T14:54:25.906 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 3 2024-08-19T14:54:25.907 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:54:25.907 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-08-19T14:54:25.907 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:54:25.907 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:54:25.907 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:54:25.907 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:54:25.907 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:54:25.907 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:54:25.907 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:54:25.907 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:54:25.907 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:54:25.907 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:54:25.907 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:54:25.908 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:54:25.908 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:25.908 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:25.908 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-08-19T14:54:25.908 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-08-19T14:54:25.908 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T14:54:25.908 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T14:54:26.014 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_mantu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:54:26.014 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:54:26.014 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:54:26.014 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:54:26.014 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:54:26.014 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:54:26.014 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:54:26.014 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:26.014 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:26.015 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-08-19T14:54:26.015 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 3 2024-08-19T14:54:26.015 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:54:26.015 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-08-19T14:54:26.015 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:54:26.015 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:54:26.015 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:54:26.015 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:54:26.015 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:54:26.015 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:54:26.015 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:54:26.015 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:54:26.015 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:54:26.016 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:54:26.016 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:54:26.016 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:54:26.016 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:26.016 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:26.016 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-08-19T14:54:26.016 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-08-19T14:54:26.016 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T14:54:26.016 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T14:54:27.069 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_makking_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T14:54:27.070 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T14:54:27.070 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T14:54:27.070 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T14:54:27.070 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T14:54:27.070 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T14:54:27.070 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T14:54:27.070 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-19T14:54:27.070 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:54:27.070 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-19T14:54:27.070 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:54:27.070 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:54:27.071 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:54:27.071 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:54:27.071 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:54:27.071 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:54:27.071 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:54:27.071 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:54:27.071 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:54:27.071 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:54:27.071 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:27.071 INFO:tasks.workunit.client.0.smithi102.stderr:216701: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:27.071 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:54:27.071 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:54:27.072 INFO:tasks.workunit.client.0.smithi102.stderr:216701: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:54:27.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:54:27.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:54:27.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 216700 2024-08-19T14:54:27.173 INFO:tasks.workunit.client.0.smithi102.stderr:ing_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T14:54:27.173 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T14:54:27.173 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T14:54:27.173 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T14:54:27.173 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T14:54:27.173 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T14:54:27.174 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T14:54:27.174 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-19T14:54:27.174 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:54:27.174 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-19T14:54:27.174 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:54:27.174 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:54:27.174 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:54:27.174 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:54:27.174 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:54:27.174 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:54:27.174 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:54:27.175 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:54:27.175 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:54:27.175 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:54:27.175 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:27.175 INFO:tasks.workunit.client.0.smithi102.stderr:216703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:27.175 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:54:27.175 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:54:27.175 INFO:tasks.workunit.client.0.smithi102.stderr:216703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:54:27.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:54:27.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-08-19T14:54:27.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-08-19T14:54:27.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-08-19T14:54:27.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-08-19T14:54:27.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-08-19T14:54:27.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-08-19T14:54:27.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T14:54:27.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T14:54:27.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:54:27.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:54:27.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-08-19T14:54:27.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-08-19T14:54:27.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-08-19T14:54:27.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-08-19T14:54:27.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:54:27.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:54:27.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:54:27.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:54:27.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T14:53:50.476942+0000 2024-08-19T14:54:27.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-08-19T14:54:27.960 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-08-19T14:54:27.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-08-19T14:53:50.476942+0000 2024-08-19T14:54:27.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T14:54:27.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:53:50.476942+0000 2024-08-19T14:54:27.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:54:27.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:54:27.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:54:27.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:54:27.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:54:27.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:54:27.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:54:27.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:54:28.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:53:50.476942+0000 '>' 2024-08-19T14:53:50.476942+0000 2024-08-19T14:54:28.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:54:29.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:54:29.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:54:29.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:54:29.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:54:29.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:54:29.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:54:29.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:54:29.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:53:50.476942+0000 '>' 2024-08-19T14:53:50.476942+0000 2024-08-19T14:54:29.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:54:30.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:54:30.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:54:30.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:54:30.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:54:30.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:54:30.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:54:30.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:54:30.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:54:28.504593+0000 '>' 2024-08-19T14:53:50.476942+0000 2024-08-19T14:54:30.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:54:30.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-08-19T14:54:30.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-08-19T14:54:30.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:54:30.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:54:30.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 220396"' 2024-08-19T14:54:30.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 220396' 2024-08-19T14:54:30.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T14:54:30.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:54:30.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:54:30.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 220398"' 2024-08-19T14:54:30.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-08-19T14:54:30.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 220398' 2024-08-19T14:54:30.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-08-19T14:54:30.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 220396 220398' 2024-08-19T14:54:30.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-08-19T14:54:30.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:54:30.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 220396 2024-08-19T14:54:30.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/220399: /' 2024-08-19T14:54:30.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T14:54:30.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/220401: /' 2024-08-19T14:54:33.593 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:54:33.593 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:54:33.593 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-08-19T14:54:33.593 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:54:33.593 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T14:54:33.594 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:54:33.594 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:54:33.594 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-08-19T14:54:33.594 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:54:33.594 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-08-19T14:54:33.594 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:54:33.594 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:54:33.594 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:54:33.594 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:54:33.594 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:54:33.594 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:54:33.594 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T14:54:33.595 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:54:33.595 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:54:33.595 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-08-19T14:54:33.595 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:54:33.595 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-08-19T14:54:33.595 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-08-19T14:54:33.595 INFO:tasks.workunit.client.0.smithi102.stderr:220401: _ 2024-08-19T14:54:33.595 INFO:tasks.workunit.client.0.smithi102.stderr:220401: hinfo_key 2024-08-19T14:54:33.595 INFO:tasks.workunit.client.0.smithi102.stderr:220401: snapset 2024-08-19T14:54:33.595 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-08-19T14:54:33.595 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:54:33.595 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:54:33.596 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-08-19T14:54:33.596 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:54:33.596 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-08-19T14:54:33.596 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:54:33.596 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:54:33.596 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:54:33.596 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:54:33.596 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-08-19T14:54:33.596 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-08-19T14:54:33.597 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:54:33.597 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:54:33.597 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:54:33.597 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:54:33.597 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:54:33.597 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:54:33.597 INFO:tasks.workunit.client.0.smithi102.stderr:220401: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:54:33.597 INFO:tasks.workunit.client.0.smithi102.stderr:220401: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:54:33.597 INFO:tasks.workunit.client.0.smithi102.stderr:220401: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:54:33.597 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:54:33.597 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:54:33.597 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:54:33.597 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:54:33.598 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:54:33.598 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:54:33.598 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:54:33.598 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:54:33.598 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:54:33.598 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:54:33.598 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:54:33.598 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-08-19T14:54:33.598 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-08-19T14:54:33.598 INFO:tasks.workunit.client.0.smithi102.stderr:220401: start osd.9 2024-08-19T14:54:33.711 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:54:33.711 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:54:33.711 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-08-19T14:54:33.711 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:54:33.712 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-08-19T14:54:33.712 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:54:33.712 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:54:33.712 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-08-19T14:54:33.712 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:54:33.712 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-08-19T14:54:33.712 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:54:33.712 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:54:33.712 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:54:33.712 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:54:33.712 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:54:33.712 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:54:33.712 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-08-19T14:54:33.713 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:54:33.713 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:54:33.713 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-08-19T14:54:33.713 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:54:33.713 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-08-19T14:54:33.713 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-08-19T14:54:33.713 INFO:tasks.workunit.client.0.smithi102.stderr:220399: _ 2024-08-19T14:54:33.713 INFO:tasks.workunit.client.0.smithi102.stderr:220399: hinfo_key 2024-08-19T14:54:33.713 INFO:tasks.workunit.client.0.smithi102.stderr:220399: snapset 2024-08-19T14:54:33.713 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-08-19T14:54:33.713 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:54:33.713 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:54:33.713 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-08-19T14:54:33.714 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:54:33.714 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-08-19T14:54:33.714 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:54:33.714 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:54:33.714 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:54:33.714 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:54:33.714 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-08-19T14:54:33.714 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-08-19T14:54:33.714 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:54:33.715 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:54:33.715 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:54:33.715 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:54:33.715 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:54:33.715 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:54:33.715 INFO:tasks.workunit.client.0.smithi102.stderr:220399: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:54:33.715 INFO:tasks.workunit.client.0.smithi102.stderr:220399: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:54:33.715 INFO:tasks.workunit.client.0.smithi102.stderr:220399: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:54:33.715 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:54:33.715 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:54:33.715 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:54:33.715 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:54:33.715 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:54:33.716 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:54:33.716 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:54:33.716 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:54:33.716 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:54:33.716 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:54:33.716 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:54:33.716 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-08-19T14:54:33.716 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-08-19T14:54:33.716 INFO:tasks.workunit.client.0.smithi102.stderr:220399: start osd.5 2024-08-19T14:54:38.343 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:54:38.344 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-08-19T14:54:38.344 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-08-19T14:54:38.344 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:54:38.344 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:54:38.344 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:54:38.344 INFO:tasks.workunit.client.0.smithi102.stderr:220401: 2024-08-19T14:54:33.626+0000 7fdd58822640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:38.344 INFO:tasks.workunit.client.0.smithi102.stderr:220401: 2024-08-19T14:54:33.628+0000 7fdd58822640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:38.344 INFO:tasks.workunit.client.0.smithi102.stderr:220401: 2024-08-19T14:54:33.629+0000 7fdd58822640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:38.344 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-08-19T14:54:38.344 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:54:38.344 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-08-19T14:54:38.345 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:54:38.345 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:54:38.345 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:38.345 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:54:38.345 INFO:tasks.workunit.client.0.smithi102.stderr:220401: 0 2024-08-19T14:54:38.345 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:38.345 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:54:38.345 INFO:tasks.workunit.client.0.smithi102.stderr:220401: 2024-08-19T14:54:34.155+0000 7fdd58822640 -1 Falling back to public interface 2024-08-19T14:54:38.345 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:38.345 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:38.345 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:38.346 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:54:38.346 INFO:tasks.workunit.client.0.smithi102.stderr:220401: 1 2024-08-19T14:54:38.346 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:38.346 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:54:38.346 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:38.346 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:38.346 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:38.346 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:54:38.346 INFO:tasks.workunit.client.0.smithi102.stderr:220401: 2 2024-08-19T14:54:38.346 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:38.346 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:54:38.346 INFO:tasks.workunit.client.0.smithi102.stderr:220401: 2024-08-19T14:54:36.751+0000 7fdd58822640 -1 osd.9 99 log_to_monitors true 2024-08-19T14:54:38.346 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:38.347 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:38.347 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:38.347 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:54:38.347 INFO:tasks.workunit.client.0.smithi102.stderr:220401: 3 2024-08-19T14:54:38.347 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:38.347 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T14:54:38.347 INFO:tasks.workunit.client.0.smithi102.stderr:220401: osd.9 up in weight 1 up_from 105 up_thru 0 down_aovery_ops 2024-08-19T14:54:38.347 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-08-19T14:54:38.347 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-08-19T14:54:38.347 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:54:38.347 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:54:38.347 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:54:38.347 INFO:tasks.workunit.client.0.smithi102.stderr:220399: 2024-08-19T14:54:33.740+0000 7feec011f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:38.347 INFO:tasks.workunit.client.0.smithi102.stderr:220399: 2024-08-19T14:54:33.742+0000 7feec011f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:38.348 INFO:tasks.workunit.client.0.smithi102.stderr:220399: 2024-08-19T14:54:33.743+0000 7feec011f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:38.348 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-08-19T14:54:38.348 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:54:38.348 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-08-19T14:54:38.348 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:54:38.348 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:54:38.348 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:38.348 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:54:38.348 INFO:tasks.workunit.client.0.smithi102.stderr:220399: 0 2024-08-19T14:54:38.348 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:54:38.348 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:38.348 INFO:tasks.workunit.client.0.smithi102.stderr:220399: 2024-08-19T14:54:34.262+0000 7feec011f640 -1 Falling back to public interface 2024-08-19T14:54:38.348 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:38.349 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:38.349 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:38.349 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:54:38.349 INFO:tasks.workunit.client.0.smithi102.stderr:220399: 1 2024-08-19T14:54:38.349 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:38.349 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:54:38.349 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:38.349 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:38.349 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:38.349 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:54:38.349 INFO:tasks.workunit.client.0.smithi102.stderr:220399: 2 2024-08-19T14:54:38.349 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:38.349 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:54:38.350 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:38.350 INFO:tasks.workunit.client.0.smithi102.stderr:220399: 2024-08-19T14:54:36.992+0000 7feec011f640 -1 osd.5 99 log_to_monitors true 2024-08-19T14:54:38.350 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:38.350 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:38.350 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:54:38.350 INFO:tasks.workunit.client.0.smithi102.stderr:220399: 3 2024-08-19T14:54:38.350 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:38.350 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:54:39.337 INFO:tasks.workunit.client.0.smithi102.stderr:220399: osd.5 up in weight 1 up_from 105 up_thru 105 downt 100 last_clean_interval [96,99) [v2:127.0.0.1:6842/905845617,v1:127.0.0.1:6843/905845617] [v2:127.0.0.1:6844/905845617,v1:127.0.0.1:6845/905845617] exists,up d4d5322d-f118-4a96-ac94-665aa58c0de8 2024-08-19T14:54:39.337 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:54:39.337 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:54:39.337 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:54:39.337 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:54:39.337 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:54:39.337 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:54:39.337 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:54:39.337 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:54:39.337 INFO:tasks.workunit.client.0.smithi102.stderr:220401: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:54:39.338 INFO:tasks.workunit.client.0.smithi102.stderr:220401: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:54:39.338 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:54:39.338 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:54:39.338 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:54:39.338 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:54:39.338 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:54:39.338 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:54:39.338 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:54:39.338 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:54:39.338 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:54:39.339 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:54:39.339 INFO:tasks.workunit.client.0.smithi102.stderr:220401: 1 2024-08-19T14:54:39.339 INFO:tasks.workunit.client.0.smithi102.stderr:220401: 2 2024-08-19T14:54:39.339 INFO:tasks.workunit.client.0.smithi102.stderr:220401: 3 2024-08-19T14:54:39.339 INFO:tasks.workunit.client.0.smithi102.stderr:220401: 4 2024-08-19T14:54:39.339 INFO:tasks.workunit.client.0.smithi102.stderr:220401: 5 2024-08-19T14:54:39.339 INFO:tasks.workunit.client.0.smithi102.stderr:220401: 6 2024-08-19T14:54:39.339 INFO:tasks.workunit.client.0.smithi102.stderr:220401: 7 2024-08-19T14:54:39.339 INFO:tasks.workunit.client.0.smithi102.stderr:220401: 8 2024-08-19T14:54:39.339 INFO:tasks.workunit.client.0.smithi102.stderr:220401: 9' 2024-08-19T14:54:39.339 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:54:39.339 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:39.340 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:54:39.340 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836581 2024-08-19T14:54:39.340 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836581 2024-08-19T14:54:39.340 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581' 2024-08-19T14:54:39.340 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:39.340 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:54:39.340 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673050 2024-08-19T14:54:39.340 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673050 2024-08-19T14:54:39.340 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673050' 2024-08-19T14:54:39.340 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:39.341 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.cli_at 100 last_clean_interval [97,99) [v2:127.0.0.1:6874/3861443697,v1:127.0.0.1:6875/3861443697] [v2:127.0.0.1:6876/3861443697,v1:127.0.0.1:6877/3861443697] exists,up 7c59bef0-d717-44f2-b3df-88b33605d1af 2024-08-19T14:54:39.341 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:54:39.341 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:54:39.341 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:54:39.341 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:54:39.341 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:54:39.341 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:54:39.341 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:54:39.341 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:54:39.342 INFO:tasks.workunit.client.0.smithi102.stderr:220399: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:54:39.342 INFO:tasks.workunit.client.0.smithi102.stderr:220399: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:54:39.342 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:54:39.342 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:54:39.342 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:54:39.342 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:54:39.342 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:54:39.342 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:54:39.342 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:54:39.343 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:54:39.343 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:54:39.343 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:54:39.343 INFO:tasks.workunit.client.0.smithi102.stderr:220399: 1 2024-08-19T14:54:39.343 INFO:tasks.workunit.client.0.smithi102.stderr:220399: 2 2024-08-19T14:54:39.343 INFO:tasks.workunit.client.0.smithi102.stderr:220399: 3 2024-08-19T14:54:39.343 INFO:tasks.workunit.client.0.smithi102.stderr:220399: 4 2024-08-19T14:54:39.343 INFO:tasks.workunit.client.0.smithi102.stderr:220399: 5 2024-08-19T14:54:39.343 INFO:tasks.workunit.client.0.smithi102.stderr:220399: 6 2024-08-19T14:54:39.343 INFO:tasks.workunit.client.0.smithi102.stderr:220399: 7 2024-08-19T14:54:39.343 INFO:tasks.workunit.client.0.smithi102.stderr:220399: 8 2024-08-19T14:54:39.343 INFO:tasks.workunit.client.0.smithi102.stderr:220399: 9' 2024-08-19T14:54:39.344 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:54:39.344 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:39.344 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:54:39.344 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836582 2024-08-19T14:54:39.344 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836582 2024-08-19T14:54:39.344 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582' 2024-08-19T14:54:39.344 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:39.344 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:54:39.344 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673051 2024-08-19T14:54:39.344 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673051 2024-08-19T14:54:39.344 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673051' 2024-08-19T14:54:39.344 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:40.182 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/cloent.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:54:40.182 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509524 2024-08-19T14:54:40.182 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509524 2024-08-19T14:54:40.183 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673050 2-64424509524' 2024-08-19T14:54:40.183 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:40.183 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:54:40.183 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710745 2024-08-19T14:54:40.183 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710745 2024-08-19T14:54:40.183 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673050 2-64424509524 3-300647710745' 2024-08-19T14:54:40.183 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:40.183 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T14:54:40.183 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182467 2024-08-19T14:54:40.183 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182467 2024-08-19T14:54:40.183 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673050 2-64424509524 3-300647710745 4-107374182467' 2024-08-19T14:54:40.183 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:40.184 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T14:54:40.184 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566082 2024-08-19T14:54:40.184 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566082 2024-08-19T14:54:40.184 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673050 2-64424509524 3-300647710745 4-107374182467 5-450971566082' 2024-08-19T14:54:40.184 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:40.184 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T14:54:40.184 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855419 2024-08-19T14:54:40.184 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855419 2024-08-19T14:54:40.184 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673050 2-64424509524 3-300647710745 4-107374182467 5-450971566082 6-150323855419' 2024-08-19T14:54:40.184 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:40.184 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T14:54:40.184 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691892 2024-08-19T14:54:40.184 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691892 2024-08-19T14:54:40.185 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673050 2-64424509524 3-300647710745 4-107374182467 5-450971566082 6-150323855419 7-171798691892' 2024-08-19T14:54:40.185 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:40.185 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T14:54:40.185 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtene.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:54:40.185 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509525 2024-08-19T14:54:40.185 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509525 2024-08-19T14:54:40.185 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673051 2-64424509525' 2024-08-19T14:54:40.185 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:40.185 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:54:40.185 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710746 2024-08-19T14:54:40.185 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710746 2024-08-19T14:54:40.185 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673051 2-64424509525 3-300647710746' 2024-08-19T14:54:40.185 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:40.186 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T14:54:40.186 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182468 2024-08-19T14:54:40.186 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182468 2024-08-19T14:54:40.186 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673051 2-64424509525 3-300647710746 4-107374182468' 2024-08-19T14:54:40.186 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:40.186 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T14:54:40.186 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566083 2024-08-19T14:54:40.186 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566083 2024-08-19T14:54:40.186 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673051 2-64424509525 3-300647710746 4-107374182468 5-450971566083' 2024-08-19T14:54:40.186 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:40.186 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T14:54:40.186 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855418 2024-08-19T14:54:40.187 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855418 2024-08-19T14:54:40.187 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673051 2-64424509525 3-300647710746 4-107374182468 5-450971566083 6-150323855418' 2024-08-19T14:54:40.187 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:40.187 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T14:54:40.187 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691893 2024-08-19T14:54:40.187 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691893 2024-08-19T14:54:40.187 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673051 2-64424509525 3-300647710746 4-107374182468 5-450971566083 6-150323855418 7-171798691893' 2024-08-19T14:54:40.187 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:40.187 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T14:54:42.136 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/st/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528361 2024-08-19T14:54:42.136 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528361 2024-08-19T14:54:42.136 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673050 2-64424509524 3-300647710745 4-107374182467 5-450971566082 6-150323855419 7-171798691892 8-193273528361' 2024-08-19T14:54:42.136 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:42.136 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T14:54:42.136 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566082 2024-08-19T14:54:42.136 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566082 2024-08-19T14:54:42.136 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673050 2-64424509524 3-300647710745 4-107374182467 5-450971566082 6-150323855419 7-171798691892 8-193273528361 9-450971566082' 2024-08-19T14:54:42.137 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:42.137 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836581 2024-08-19T14:54:42.137 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:42.137 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:54:42.137 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836581 2024-08-19T14:54:42.137 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:42.137 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836581 2024-08-19T14:54:42.137 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-08-19T14:54:42.137 INFO:tasks.workunit.client.0.smithi102.stderr:220401: waiting osd.0 seq 21474836581 2024-08-19T14:54:42.137 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:54:42.138 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836579 -lt 21474836581 2024-08-19T14:54:42.138 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:54:42.138 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:54:42.138 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:54:42.138 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836582 -lt 21474836581 2024-08-19T14:54:42.138 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:42.138 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673050 2024-08-19T14:54:42.138 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:42.138 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:54:42.138 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673050 2024-08-19T14:54:42.139 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:42.139 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673050 2024-08-19T14:54:42.139 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673050' 2024-08-19T14:54:42.139 INFO:tasks.workunit.client.0.smithi102.stderr:220401: waiting osd.1 seq 42949673050 2024-08-19T14:54:42.139 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:54:42.139 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673051 -lt 42949673050 2024-08-19T14:54:42.210 INFO:tasks.workunit.client.0.smithi102.stderr:2cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528362 2024-08-19T14:54:42.211 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528362 2024-08-19T14:54:42.211 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673051 2-64424509525 3-300647710746 4-107374182468 5-450971566083 6-150323855418 7-171798691893 8-193273528362' 2024-08-19T14:54:42.211 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:42.211 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T14:54:42.211 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566083 2024-08-19T14:54:42.211 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566083 2024-08-19T14:54:42.211 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673051 2-64424509525 3-300647710746 4-107374182468 5-450971566083 6-150323855418 7-171798691893 8-193273528362 9-450971566083' 2024-08-19T14:54:42.211 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:42.211 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836582 2024-08-19T14:54:42.211 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:42.211 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:54:42.212 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836582 2024-08-19T14:54:42.212 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:42.212 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836582 2024-08-19T14:54:42.212 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2024-08-19T14:54:42.212 INFO:tasks.workunit.client.0.smithi102.stderr:220399: waiting osd.0 seq 21474836582 2024-08-19T14:54:42.212 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:54:42.212 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836579 -lt 21474836582 2024-08-19T14:54:42.212 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:54:42.212 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:54:42.212 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:54:42.212 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836582 -lt 21474836582 2024-08-19T14:54:42.212 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:42.212 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673051 2024-08-19T14:54:42.213 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:42.213 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:54:42.213 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673051 2024-08-19T14:54:42.213 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:42.213 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673051 2024-08-19T14:54:42.213 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-08-19T14:54:42.213 INFO:tasks.workunit.client.0.smithi102.stderr:220399: waiting osd.1 seq 42949673051 2024-08-19T14:54:42.213 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:54:43.083 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673051 -lt 429496720401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:43.084 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509524 2024-08-19T14:54:43.084 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:43.084 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:54:43.084 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509524 2024-08-19T14:54:43.084 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:43.084 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509524 2024-08-19T14:54:43.084 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-08-19T14:54:43.084 INFO:tasks.workunit.client.0.smithi102.stderr:220401: waiting osd.2 seq 64424509524 2024-08-19T14:54:43.084 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:54:43.084 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509525 -lt 64424509524 2024-08-19T14:54:43.084 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:43.084 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710745 2024-08-19T14:54:43.084 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:54:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710745 2024-08-19T14:54:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710745 2024-08-19T14:54:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710745' 2024-08-19T14:54:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:220401: waiting osd.3 seq 300647710745 2024-08-19T14:54:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:54:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710747 -lt 300647710745 2024-08-19T14:54:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182467 2024-08-19T14:54:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T14:54:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182467 2024-08-19T14:54:43.086 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:43.086 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182467 2024-08-19T14:54:43.086 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182467' 2024-08-19T14:54:43.086 INFO:tasks.workunit.client.0.smithi102.stderr:220401: waiting osd.4 seq 107374182467 2024-08-19T14:54:43.086 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T14:54:43.086 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182469 -lt 107374182467 2024-08-19T14:54:43.086 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:43.086 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-450971566082 2024-08-19T14:54:43.086 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:43.168 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:3051 2024-08-19T14:54:43.168 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:43.168 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509525 2024-08-19T14:54:43.168 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:43.168 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:54:43.168 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509525 2024-08-19T14:54:43.168 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:43.169 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509525 2024-08-19T14:54:43.169 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2024-08-19T14:54:43.169 INFO:tasks.workunit.client.0.smithi102.stderr:220399: waiting osd.2 seq 64424509525 2024-08-19T14:54:43.169 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:54:43.169 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509525 -lt 64424509525 2024-08-19T14:54:43.169 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:43.169 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710746 2024-08-19T14:54:43.169 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:43.169 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:54:43.169 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710746 2024-08-19T14:54:43.169 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:43.170 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710746 2024-08-19T14:54:43.170 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710746' 2024-08-19T14:54:43.170 INFO:tasks.workunit.client.0.smithi102.stderr:220399: waiting osd.3 seq 300647710746 2024-08-19T14:54:43.170 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:54:43.170 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710747 -lt 300647710746 2024-08-19T14:54:43.170 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:43.170 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182468 2024-08-19T14:54:43.170 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:43.170 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T14:54:43.170 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182468 2024-08-19T14:54:43.171 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:43.171 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182468 2024-08-19T14:54:43.171 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182468' 2024-08-19T14:54:43.171 INFO:tasks.workunit.client.0.smithi102.stderr:220399: waiting osd.4 seq 107374182468 2024-08-19T14:54:43.171 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T14:54:43.171 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182469 -lt 107374182468 2024-08-19T14:54:43.171 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:43.171 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-450971566083 2024-08-19T14:54:43.171 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:43.991 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpe2211: flush_pg_stats: osd=5 2024-08-19T14:54:43.991 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-450971566082 2024-08-19T14:54:43.991 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:43.991 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566082 2024-08-19T14:54:43.991 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 450971566082' 2024-08-19T14:54:43.991 INFO:tasks.workunit.client.0.smithi102.stderr:220401: waiting osd.5 seq 450971566082 2024-08-19T14:54:43.992 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T14:54:43.992 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566083 -lt 450971566082 2024-08-19T14:54:43.992 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:43.992 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855419 2024-08-19T14:54:43.992 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:43.992 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T14:54:43.992 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855419 2024-08-19T14:54:43.992 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:43.992 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855419 2024-08-19T14:54:43.992 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855419' 2024-08-19T14:54:43.992 INFO:tasks.workunit.client.0.smithi102.stderr:220401: waiting osd.6 seq 150323855419 2024-08-19T14:54:43.993 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T14:54:43.993 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855420 -lt 150323855419 2024-08-19T14:54:43.993 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:43.993 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691892 2024-08-19T14:54:43.993 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:43.993 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T14:54:43.993 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691892 2024-08-19T14:54:43.993 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:43.993 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691892 2024-08-19T14:54:43.993 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691892' 2024-08-19T14:54:43.993 INFO:tasks.workunit.client.0.smithi102.stderr:220401: waiting osd.7 seq 171798691892 2024-08-19T14:54:43.994 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T14:54:43.994 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691893 -lt 171798691892 2024-08-19T14:54:43.994 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:43.994 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528361 2024-08-19T14:54:43.994 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:43.994 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T14:54:43.994 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528361 2024-08-19T14:54:43.994 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:43.994 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528361 2024-08-19T14:54:44.074 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.rs.sh:2211: flush_pg_stats: osd=5 2024-08-19T14:54:44.074 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-450971566083 2024-08-19T14:54:44.074 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:44.074 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566083 2024-08-19T14:54:44.074 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 450971566083' 2024-08-19T14:54:44.074 INFO:tasks.workunit.client.0.smithi102.stderr:220399: waiting osd.5 seq 450971566083 2024-08-19T14:54:44.074 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T14:54:44.075 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566083 -lt 450971566083 2024-08-19T14:54:44.075 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:44.075 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855418 2024-08-19T14:54:44.075 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:44.075 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T14:54:44.075 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855418 2024-08-19T14:54:44.075 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:44.075 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855418 2024-08-19T14:54:44.075 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855418' 2024-08-19T14:54:44.075 INFO:tasks.workunit.client.0.smithi102.stderr:220399: waiting osd.6 seq 150323855418 2024-08-19T14:54:44.075 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T14:54:44.075 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855420 -lt 150323855418 2024-08-19T14:54:44.075 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:44.076 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691893 2024-08-19T14:54:44.076 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:44.076 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T14:54:44.076 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691893 2024-08-19T14:54:44.076 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:44.076 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691893 2024-08-19T14:54:44.076 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691893' 2024-08-19T14:54:44.076 INFO:tasks.workunit.client.0.smithi102.stderr:220399: waiting osd.7 seq 171798691893 2024-08-19T14:54:44.076 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T14:54:44.076 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691893 -lt 171798691893 2024-08-19T14:54:44.076 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:44.077 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528362 2024-08-19T14:54:44.077 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:44.077 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T14:54:44.077 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528362 2024-08-19T14:54:44.077 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:44.077 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528362 2024-08-19T14:54:45.611 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528361' 2024-08-19T14:54:45.611 INFO:tasks.workunit.client.0.smithi102.stderr:220401: waiting osd.8 seq 193273528361 2024-08-19T14:54:45.611 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T14:54:45.611 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528362 -lt 193273528361 2024-08-19T14:54:45.612 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:45.612 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566082 2024-08-19T14:54:45.612 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:45.612 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T14:54:45.612 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566082 2024-08-19T14:54:45.612 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:45.612 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566082 2024-08-19T14:54:45.612 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566082' 2024-08-19T14:54:45.612 INFO:tasks.workunit.client.0.smithi102.stderr:220401: waiting osd.9 seq 450971566082 2024-08-19T14:54:45.612 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T14:54:45.612 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566083 -lt 450971566082 2024-08-19T14:54:45.612 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:54:45.612 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:45.612 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:45.613 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:54:45.613 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:54:45.613 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:54:45.613 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:54:45.613 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:54:45.613 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:54:45.613 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:54:45.613 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:54:45.613 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:54:45.613 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:54:45.613 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:45.613 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:45.614 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-08-19T14:54:45.614 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-08-19T14:54:45.614 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:54:45.614 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-08-19T14:54:45.692 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sclone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528362' 2024-08-19T14:54:45.693 INFO:tasks.workunit.client.0.smithi102.stderr:220399: waiting osd.8 seq 193273528362 2024-08-19T14:54:45.693 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T14:54:45.693 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528362 -lt 193273528362 2024-08-19T14:54:45.693 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:45.693 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566083 2024-08-19T14:54:45.693 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:45.693 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T14:54:45.693 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566083 2024-08-19T14:54:45.693 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:45.694 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566083 2024-08-19T14:54:45.694 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566083' 2024-08-19T14:54:45.694 INFO:tasks.workunit.client.0.smithi102.stderr:220399: waiting osd.9 seq 450971566083 2024-08-19T14:54:45.694 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T14:54:45.694 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566083 -lt 450971566083 2024-08-19T14:54:45.694 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:54:45.694 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:45.694 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:45.694 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:54:45.694 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:54:45.694 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:54:45.694 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:54:45.695 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:54:45.695 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:54:45.695 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:54:45.695 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:54:45.695 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:54:45.695 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:54:45.695 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:45.695 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:45.695 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-08-19T14:54:45.695 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-08-19T14:54:45.695 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T14:54:45.695 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-08-19T14:54:46.901 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helh:1669: wait_for_clean: eval 2024-08-19T14:54:46.902 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:54:46.902 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:54:46.902 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:54:46.902 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:54:46.902 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:54:46.902 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:54:46.902 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:54:46.902 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:54:46.902 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:54:46.902 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:54:46.903 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:54:46.903 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:46.903 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:46.903 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-08-19T14:54:46.903 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-08-19T14:54:46.903 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T14:54:46.903 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T14:54:46.903 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T14:54:46.903 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T14:54:46.903 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T14:54:46.903 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T14:54:46.903 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T14:54:46.903 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T14:54:46.904 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T14:54:46.904 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-19T14:54:46.904 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:54:46.904 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-19T14:54:46.904 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:54:46.904 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:54:46.904 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:54:46.994 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_apers.sh:1669: wait_for_clean: eval 2024-08-19T14:54:46.994 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T14:54:46.994 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:54:46.994 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:54:46.995 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:54:46.995 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:54:46.995 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:54:46.995 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:54:46.995 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:54:46.995 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:54:46.995 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T14:54:46.995 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:54:46.995 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:46.995 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:46.995 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-08-19T14:54:46.995 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-08-19T14:54:46.996 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T14:54:46.996 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T14:54:46.996 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T14:54:46.996 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T14:54:46.996 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T14:54:46.996 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T14:54:46.996 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T14:54:46.996 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T14:54:46.996 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T14:54:46.996 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-19T14:54:46.996 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T14:54:46.996 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-19T14:54:46.997 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T14:54:46.997 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:54:46.997 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:54:47.521 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: getctive_clean: local expression 2024-08-19T14:54:47.522 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:54:47.522 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:54:47.522 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:54:47.522 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:54:47.522 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:54:47.522 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:54:47.522 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:47.522 INFO:tasks.workunit.client.0.smithi102.stderr:220401: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:47.522 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:54:47.522 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:54:47.522 INFO:tasks.workunit.client.0.smithi102.stderr:220401: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:54:47.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:54:47.609 INFO:tasks.workunit.client.0.smithi102.stderr:_num_active_clean: local expression 2024-08-19T14:54:47.610 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:54:47.610 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:54:47.610 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:54:47.610 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:54:47.610 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:54:47.610 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:54:47.610 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:54:47.610 INFO:tasks.workunit.client.0.smithi102.stderr:220399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:54:47.610 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:54:47.610 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:54:47.610 INFO:tasks.workunit.client.0.smithi102.stderr:220399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:54:47.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:54:47.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:54:47.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 220398 2024-08-19T14:54:47.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-08-19T14:54:47.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-08-19T14:54:47.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-08-19T14:54:47.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-08-19T14:54:47.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-08-19T14:54:47.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T14:54:47.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T14:54:47.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-08-19T14:54:47.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-08-19T14:54:47.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-08-19T14:54:47.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-08-19T14:54:47.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-08-19T14:54:47.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-08-19T14:54:47.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:54:47.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:54:47.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:54:47.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 223919"' 2024-08-19T14:54:47.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 223919' 2024-08-19T14:54:47.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T14:54:47.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:54:47.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:54:47.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 223921"' 2024-08-19T14:54:47.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:54:47.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 223921' 2024-08-19T14:54:47.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-08-19T14:54:47.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 223919 223921' 2024-08-19T14:54:47.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-08-19T14:54:47.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:54:47.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 223919 2024-08-19T14:54:47.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/223922: /' 2024-08-19T14:54:47.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T14:54:47.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/223924: /' 2024-08-19T14:54:50.428 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:54:50.428 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:54:50.428 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-08-19T14:54:50.428 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:54:50.428 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T14:54:50.428 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:54:50.428 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:54:50.428 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-08-19T14:54:50.428 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:54:50.428 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-08-19T14:54:50.428 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:54:50.429 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:54:50.429 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:54:50.429 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:54:50.429 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:54:50.429 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:54:50.429 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T14:54:50.429 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:54:50.429 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:54:50.429 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-08-19T14:54:50.429 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:54:50.429 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-08-19T14:54:50.429 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-08-19T14:54:50.429 INFO:tasks.workunit.client.0.smithi102.stderr:223924: remove 1#2:eb822e21:::SOMETHING:head# 2024-08-19T14:54:50.430 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-08-19T14:54:50.430 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:54:50.430 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:54:50.430 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-08-19T14:54:50.430 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:54:50.430 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-08-19T14:54:50.430 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:54:50.430 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:54:50.431 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:54:50.431 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:54:50.431 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-08-19T14:54:50.431 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-08-19T14:54:50.431 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:54:50.432 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:54:50.432 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:54:50.432 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:54:50.432 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:54:50.432 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:54:50.432 INFO:tasks.workunit.client.0.smithi102.stderr:223924: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:54:50.432 INFO:tasks.workunit.client.0.smithi102.stderr:223924: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:54:50.432 INFO:tasks.workunit.client.0.smithi102.stderr:223924: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:54:50.432 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:54:50.432 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:54:50.432 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:54:50.432 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:54:50.432 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:54:50.432 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:54:50.433 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:54:50.433 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:54:50.433 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:54:50.433 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:54:50.433 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:54:50.433 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-08-19T14:54:50.433 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-08-19T14:54:50.433 INFO:tasks.workunit.client.0.smithi102.stderr:223924: start osd.5 2024-08-19T14:54:50.433 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:54:50.468 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:54:50.468 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:54:50.468 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-08-19T14:54:50.468 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:54:50.468 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:54:50.468 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:54:50.468 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:54:50.469 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-08-19T14:54:50.469 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:54:50.469 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-08-19T14:54:50.469 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:54:50.469 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:54:50.469 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:54:50.469 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:54:50.469 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:54:50.469 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:54:50.469 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T14:54:50.469 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:54:50.469 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:54:50.469 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-08-19T14:54:50.470 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:54:50.470 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:54:50.470 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-08-19T14:54:50.470 INFO:tasks.workunit.client.0.smithi102.stderr:223922: remove 0#2:eb822e21:::SOMETHING:head# 2024-08-19T14:54:50.470 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-08-19T14:54:50.470 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:54:50.470 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:54:50.470 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-19T14:54:50.470 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:54:50.470 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:54:50.470 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:54:50.471 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:54:50.471 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:54:50.471 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:54:50.471 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T14:54:50.471 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T14:54:50.471 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:54:50.471 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:54:50.471 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:54:50.472 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:54:50.472 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:54:50.472 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:54:50.472 INFO:tasks.workunit.client.0.smithi102.stderr:223922: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:54:50.472 INFO:tasks.workunit.client.0.smithi102.stderr:223922: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:54:50.472 INFO:tasks.workunit.client.0.smithi102.stderr:223922: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:54:50.472 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:54:50.472 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:54:50.472 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:54:50.472 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:54:50.472 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:54:50.473 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:54:50.473 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:54:50.473 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:54:50.473 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:54:50.473 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:54:50.473 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:54:50.473 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T14:54:50.473 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-19T14:54:50.473 INFO:tasks.workunit.client.0.smithi102.stderr:223922: start osd.3 2024-08-19T14:54:50.473 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:54:55.166 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-08-19T14:54:55.166 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-19T14:54:55.166 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:54:55.166 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:54:55.166 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:54:55.167 INFO:tasks.workunit.client.0.smithi102.stderr:223922: 2024-08-19T14:54:50.499+0000 7fcd82a16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:55.167 INFO:tasks.workunit.client.0.smithi102.stderr:223922: 2024-08-19T14:54:50.501+0000 7fcd82a16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:55.167 INFO:tasks.workunit.client.0.smithi102.stderr:223922: 2024-08-19T14:54:50.502+0000 7fcd82a16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:55.167 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-19T14:54:55.167 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:54:55.167 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T14:54:55.167 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:54:55.167 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:54:55.167 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:55.167 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:54:55.167 INFO:tasks.workunit.client.0.smithi102.stderr:223922: 0 2024-08-19T14:54:55.167 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:55.167 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:54:55.168 INFO:tasks.workunit.client.0.smithi102.stderr:223922: 2024-08-19T14:54:51.017+0000 7fcd82a16640 -1 Falling back to public interface 2024-08-19T14:54:55.168 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:55.168 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:55.168 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:55.168 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:54:55.168 INFO:tasks.workunit.client.0.smithi102.stderr:223922: 1 2024-08-19T14:54:55.168 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:55.168 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:54:55.168 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:55.168 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:55.168 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:55.168 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:54:55.168 INFO:tasks.workunit.client.0.smithi102.stderr:223922: 2 2024-08-19T14:54:55.169 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:55.169 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:54:55.169 INFO:tasks.workunit.client.0.smithi102.stderr:223922: 2024-08-19T14:54:53.598+0000 7fcd82a16640 -1 osd.3 107 log_to_monitors true 2024-08-19T14:54:55.169 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:55.169 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:55.169 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:55.169 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:54:55.169 INFO:tasks.workunit.client.0.smithi102.stderr:223922: 3 2024-08-19T14:54:55.169 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:55.169 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:54:55.169 INFO:tasks.workunit.client.0.smithi102.stderr:223922: osd.3 up in weight 1 up_from 112 up_thru 106 down_at 108 l223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-08-19T14:54:55.169 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-08-19T14:54:55.169 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:54:55.169 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:54:55.170 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:54:55.170 INFO:tasks.workunit.client.0.smithi102.stderr:223924: 2024-08-19T14:54:50.460+0000 7ff633cdb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:55.170 INFO:tasks.workunit.client.0.smithi102.stderr:223924: 2024-08-19T14:54:50.463+0000 7ff633cdb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:55.170 INFO:tasks.workunit.client.0.smithi102.stderr:223924: 2024-08-19T14:54:50.464+0000 7ff633cdb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:54:55.170 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-08-19T14:54:55.170 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:54:55.170 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-08-19T14:54:55.170 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:54:55.170 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:54:55.171 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:55.171 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:54:55.171 INFO:tasks.workunit.client.0.smithi102.stderr:223924: 0 2024-08-19T14:54:55.171 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:55.171 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:54:55.171 INFO:tasks.workunit.client.0.smithi102.stderr:223924: 2024-08-19T14:54:50.978+0000 7ff633cdb640 -1 Falling back to public interface 2024-08-19T14:54:55.171 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:55.171 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:55.171 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:55.171 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:54:55.171 INFO:tasks.workunit.client.0.smithi102.stderr:223924: 1 2024-08-19T14:54:55.171 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:55.171 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:54:55.171 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:55.172 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:55.172 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:55.172 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:54:55.172 INFO:tasks.workunit.client.0.smithi102.stderr:223924: 2 2024-08-19T14:54:55.172 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:55.172 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:54:55.172 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:54:55.172 INFO:tasks.workunit.client.0.smithi102.stderr:223924: 2024-08-19T14:54:53.724+0000 7ff633cdb640 -1 osd.5 107 log_to_monitors true 2024-08-19T14:54:55.172 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:54:55.172 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:54:55.172 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:54:55.172 INFO:tasks.workunit.client.0.smithi102.stderr:223924: 3 2024-08-19T14:54:55.172 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:54:55.172 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:54:56.193 INFO:tasks.workunit.client.0.smithi102.stderr:223924: osd.5 up in weight 1 up_from 112 up_thru 105 down_at 108 last_clean_interval [105,107) [v2:127.0.0.1:6826/1896638020,v1:127.0.0.1:6827/1896638020] [v2:127.0.0.1:6828/1896638020,v1:127.0.0.1:6829/1896638020] exists,up 7c59bef0-d717-44f2-b3df-88b33605d1af 2024-08-19T14:54:56.193 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:54:56.193 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:54:56.193 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:54:56.193 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:54:56.194 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:54:56.194 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:54:56.194 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:54:56.194 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:54:56.194 INFO:tasks.workunit.client.0.smithi102.stderr:223924: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:54:56.194 INFO:tasks.workunit.client.0.smithi102.stderr:223924: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:54:56.194 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:54:56.194 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:54:56.194 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:54:56.194 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:54:56.194 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:54:56.194 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:54:56.194 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:54:56.195 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:54:56.195 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:54:56.195 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:54:56.195 INFO:tasks.workunit.client.0.smithi102.stderr:223924: 1 2024-08-19T14:54:56.195 INFO:tasks.workunit.client.0.smithi102.stderr:223924: 2 2024-08-19T14:54:56.195 INFO:tasks.workunit.client.0.smithi102.stderr:223924: 3 2024-08-19T14:54:56.195 INFO:tasks.workunit.client.0.smithi102.stderr:223924: 4 2024-08-19T14:54:56.195 INFO:tasks.workunit.client.0.smithi102.stderr:223924: 5 2024-08-19T14:54:56.195 INFO:tasks.workunit.client.0.smithi102.stderr:223924: 6 2024-08-19T14:54:56.195 INFO:tasks.workunit.client.0.smithi102.stderr:223924: 7 2024-08-19T14:54:56.195 INFO:tasks.workunit.client.0.smithi102.stderr:223924: 8 2024-08-19T14:54:56.195 INFO:tasks.workunit.client.0.smithi102.stderr:223924: 9' 2024-08-19T14:54:56.195 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:54:56.196 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:56.196 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:54:56.196 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836586 2024-08-19T14:54:56.196 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836586 2024-08-19T14:54:56.196 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586' 2024-08-19T14:54:56.196 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:56.196 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:54:56.196 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673055 2024-08-19T14:54:56.196 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673055 2024-08-19T14:54:56.196 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673055' 2024-08-19T14:54:56.196 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:56.196 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.clieast_clean_interval [70,107) [v2:127.0.0.1:6874/2656720668,v1:127.0.0.1:6875/2656720668] [v2:127.0.0.1:6876/2656720668,v1:127.0.0.1:6877/2656720668] exists,up 0a924bc7-1b05-4420-8781-adc75bdf85fc 2024-08-19T14:54:56.196 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:54:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:54:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:54:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:54:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:54:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:54:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:54:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:54:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:223922: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:54:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:223922: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:54:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:54:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:54:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:54:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:54:56.198 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:54:56.198 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:54:56.198 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:54:56.198 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:54:56.198 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:54:56.198 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:54:56.198 INFO:tasks.workunit.client.0.smithi102.stderr:223922: 1 2024-08-19T14:54:56.198 INFO:tasks.workunit.client.0.smithi102.stderr:223922: 2 2024-08-19T14:54:56.198 INFO:tasks.workunit.client.0.smithi102.stderr:223922: 3 2024-08-19T14:54:56.198 INFO:tasks.workunit.client.0.smithi102.stderr:223922: 4 2024-08-19T14:54:56.198 INFO:tasks.workunit.client.0.smithi102.stderr:223922: 5 2024-08-19T14:54:56.198 INFO:tasks.workunit.client.0.smithi102.stderr:223922: 6 2024-08-19T14:54:56.198 INFO:tasks.workunit.client.0.smithi102.stderr:223922: 7 2024-08-19T14:54:56.199 INFO:tasks.workunit.client.0.smithi102.stderr:223922: 8 2024-08-19T14:54:56.199 INFO:tasks.workunit.client.0.smithi102.stderr:223922: 9' 2024-08-19T14:54:56.199 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:54:56.199 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:56.199 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:54:56.199 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836587 2024-08-19T14:54:56.199 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836587 2024-08-19T14:54:56.199 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587' 2024-08-19T14:54:56.199 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:56.199 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:54:56.199 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673056 2024-08-19T14:54:56.199 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673056 2024-08-19T14:54:56.199 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673056' 2024-08-19T14:54:56.199 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:57.005 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:54:57.006 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509530 2024-08-19T14:54:57.006 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509530 2024-08-19T14:54:57.006 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673056 2-64424509530' 2024-08-19T14:54:57.006 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:57.006 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:54:57.006 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=481036337155 2024-08-19T14:54:57.006 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 481036337155 2024-08-19T14:54:57.006 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673056 2-64424509530 3-481036337155' 2024-08-19T14:54:57.006 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:57.006 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T14:54:57.006 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182473 2024-08-19T14:54:57.007 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182473 2024-08-19T14:54:57.007 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673056 2-64424509530 3-481036337155 4-107374182473' 2024-08-19T14:54:57.007 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:57.007 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T14:54:57.007 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=481036337155 2024-08-19T14:54:57.007 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 481036337155 2024-08-19T14:54:57.007 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673056 2-64424509530 3-481036337155 4-107374182473 5-481036337155' 2024-08-19T14:54:57.007 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:57.007 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T14:54:57.007 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855423 2024-08-19T14:54:57.007 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855423 2024-08-19T14:54:57.007 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673056 2-64424509530 3-481036337155 4-107374182473 5-481036337155 6-150323855423' 2024-08-19T14:54:57.007 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:57.008 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T14:54:57.008 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691897 2024-08-19T14:54:57.008 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691897 2024-08-19T14:54:57.008 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673056 2-64424509530 3-481036337155 4-107374182473 5-481036337155 6-150323855423 7-171798691897' 2024-08-19T14:54:57.008 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:57.008 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T14:54:57.008 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtestnt.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:54:57.008 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509529 2024-08-19T14:54:57.008 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509529 2024-08-19T14:54:57.008 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673055 2-64424509529' 2024-08-19T14:54:57.008 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:57.008 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:54:57.008 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=481036337154 2024-08-19T14:54:57.009 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 481036337154 2024-08-19T14:54:57.009 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673055 2-64424509529 3-481036337154' 2024-08-19T14:54:57.009 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:57.009 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T14:54:57.009 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182474 2024-08-19T14:54:57.009 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182474 2024-08-19T14:54:57.009 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673055 2-64424509529 3-481036337154 4-107374182474' 2024-08-19T14:54:57.009 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:57.009 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T14:54:57.009 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=481036337154 2024-08-19T14:54:57.009 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 481036337154 2024-08-19T14:54:57.009 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673055 2-64424509529 3-481036337154 4-107374182474 5-481036337154' 2024-08-19T14:54:57.009 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:57.010 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T14:54:57.010 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855424 2024-08-19T14:54:57.010 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855424 2024-08-19T14:54:57.010 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673055 2-64424509529 3-481036337154 4-107374182474 5-481036337154 6-150323855424' 2024-08-19T14:54:57.010 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:57.010 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T14:54:57.010 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691898 2024-08-19T14:54:57.010 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691898 2024-08-19T14:54:57.010 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673055 2-64424509529 3-481036337154 4-107374182474 5-481036337154 6-150323855424 7-171798691898' 2024-08-19T14:54:57.010 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:57.010 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T14:54:57.674 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528367 2024-08-19T14:54:57.674 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528367 2024-08-19T14:54:57.675 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673055 2-64424509529 3-481036337154 4-107374182474 5-481036337154 6-150323855424 7-171798691898 8-193273528367' 2024-08-19T14:54:57.675 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:57.675 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T14:54:57.675 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566087 2024-08-19T14:54:57.675 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566087 2024-08-19T14:54:57.675 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673055 2-64424509529 3-481036337154 4-107374182474 5-481036337154 6-150323855424 7-171798691898 8-193273528367 9-450971566087' 2024-08-19T14:54:57.675 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:57.675 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836586 2024-08-19T14:54:57.675 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:57.675 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:54:57.675 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836586 2024-08-19T14:54:57.675 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:57.675 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836586 2024-08-19T14:54:57.676 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836586' 2024-08-19T14:54:57.676 INFO:tasks.workunit.client.0.smithi102.stderr:223924: waiting osd.0 seq 21474836586 2024-08-19T14:54:57.676 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:54:57.676 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836587 -lt 21474836586 2024-08-19T14:54:57.676 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:57.676 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673055 2024-08-19T14:54:57.676 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:57.676 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:54:57.676 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673055 2024-08-19T14:54:57.676 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:57.676 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673055 2024-08-19T14:54:57.676 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-08-19T14:54:57.676 INFO:tasks.workunit.client.0.smithi102.stderr:223924: waiting osd.1 seq 42949673055 2024-08-19T14:54:57.676 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:54:57.677 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673057 -lt 42949673055 2024-08-19T14:54:57.677 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:57.677 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509529 2024-08-19T14:54:57.677 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:57.677 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:54:57.724 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest//clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528366 2024-08-19T14:54:57.725 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528366 2024-08-19T14:54:57.725 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673056 2-64424509530 3-481036337155 4-107374182473 5-481036337155 6-150323855423 7-171798691897 8-193273528366' 2024-08-19T14:54:57.725 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:54:57.725 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T14:54:57.725 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566088 2024-08-19T14:54:57.725 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566088 2024-08-19T14:54:57.725 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673056 2-64424509530 3-481036337155 4-107374182473 5-481036337155 6-150323855423 7-171798691897 8-193273528366 9-450971566088' 2024-08-19T14:54:57.726 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:57.726 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836587 2024-08-19T14:54:57.726 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:57.726 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:54:57.726 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836587 2024-08-19T14:54:57.726 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:57.727 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836587 2024-08-19T14:54:57.727 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2024-08-19T14:54:57.727 INFO:tasks.workunit.client.0.smithi102.stderr:223922: waiting osd.0 seq 21474836587 2024-08-19T14:54:57.727 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:54:57.727 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836587 -lt 21474836587 2024-08-19T14:54:57.727 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:57.727 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673056 2024-08-19T14:54:57.728 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:57.728 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:54:57.728 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673056 2024-08-19T14:54:57.728 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:57.728 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673056 2024-08-19T14:54:57.728 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2024-08-19T14:54:57.729 INFO:tasks.workunit.client.0.smithi102.stderr:223922: waiting osd.1 seq 42949673056 2024-08-19T14:54:57.729 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:54:57.729 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673057 -lt 42949673056 2024-08-19T14:54:57.729 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:57.729 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509530 2024-08-19T14:54:57.729 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:57.729 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:54:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/cclone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509529 2024-08-19T14:54:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509529 2024-08-19T14:54:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509529' 2024-08-19T14:54:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:223924: waiting osd.2 seq 64424509529 2024-08-19T14:54:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:54:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509530 -lt 64424509529 2024-08-19T14:54:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-481036337154 2024-08-19T14:54:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:54:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-481036337154 2024-08-19T14:54:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=481036337154 2024-08-19T14:54:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 481036337154' 2024-08-19T14:54:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:223924: waiting osd.3 seq 481036337154 2024-08-19T14:54:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:54:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 481036337155 -lt 481036337154 2024-08-19T14:54:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182474 2024-08-19T14:54:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T14:54:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182474 2024-08-19T14:54:59.867 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:59.867 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182474 2024-08-19T14:54:59.867 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182474' 2024-08-19T14:54:59.867 INFO:tasks.workunit.client.0.smithi102.stderr:223924: waiting osd.4 seq 107374182474 2024-08-19T14:54:59.867 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T14:54:59.867 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182472 -lt 107374182474 2024-08-19T14:54:59.867 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:54:59.867 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:54:59.867 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T14:54:59.867 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182474 -lt 107374182474 2024-08-19T14:54:59.867 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:59.867 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-481036337154 2024-08-19T14:54:59.867 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:59.888 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shlone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509530 2024-08-19T14:54:59.888 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:59.888 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509530 2024-08-19T14:54:59.888 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-08-19T14:54:59.888 INFO:tasks.workunit.client.0.smithi102.stderr:223922: waiting osd.2 seq 64424509530 2024-08-19T14:54:59.888 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:54:59.888 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509530 -lt 64424509530 2024-08-19T14:54:59.888 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:59.888 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-481036337155 2024-08-19T14:54:59.889 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:59.889 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:54:59.889 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-481036337155 2024-08-19T14:54:59.889 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:59.889 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=481036337155 2024-08-19T14:54:59.889 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 481036337155' 2024-08-19T14:54:59.889 INFO:tasks.workunit.client.0.smithi102.stderr:223922: waiting osd.3 seq 481036337155 2024-08-19T14:54:59.889 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:54:59.890 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 481036337155 -lt 481036337155 2024-08-19T14:54:59.890 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:59.890 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182473 2024-08-19T14:54:59.890 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:54:59.890 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T14:54:59.890 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182473 2024-08-19T14:54:59.890 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:54:59.890 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182473 2024-08-19T14:54:59.890 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182473' 2024-08-19T14:54:59.891 INFO:tasks.workunit.client.0.smithi102.stderr:223922: waiting osd.4 seq 107374182473 2024-08-19T14:54:59.891 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T14:54:59.891 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182472 -lt 107374182473 2024-08-19T14:54:59.891 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T14:54:59.891 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T14:54:59.891 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T14:54:59.891 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182474 -lt 107374182473 2024-08-19T14:54:59.891 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:54:59.891 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-481036337155 2024-08-19T14:54:59.891 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:01.094 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T14:55:01.094 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-481036337155 2024-08-19T14:55:01.094 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:01.094 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=481036337155 2024-08-19T14:55:01.094 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 481036337155' 2024-08-19T14:55:01.094 INFO:tasks.workunit.client.0.smithi102.stderr:223922: waiting osd.5 seq 481036337155 2024-08-19T14:55:01.094 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T14:55:01.094 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 481036337155 -lt 481036337155 2024-08-19T14:55:01.094 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:01.094 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855423 2024-08-19T14:55:01.095 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:01.095 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T14:55:01.095 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855423 2024-08-19T14:55:01.095 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:01.095 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855423 2024-08-19T14:55:01.095 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855423' 2024-08-19T14:55:01.095 INFO:tasks.workunit.client.0.smithi102.stderr:223922: waiting osd.6 seq 150323855423 2024-08-19T14:55:01.095 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T14:55:01.095 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855425 -lt 150323855423 2024-08-19T14:55:01.095 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:01.095 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691897 2024-08-19T14:55:01.095 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:01.095 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T14:55:01.096 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691897 2024-08-19T14:55:01.096 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:01.096 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691897 2024-08-19T14:55:01.096 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691897' 2024-08-19T14:55:01.096 INFO:tasks.workunit.client.0.smithi102.stderr:223922: waiting osd.7 seq 171798691897 2024-08-19T14:55:01.096 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T14:55:01.096 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691899 -lt 171798691897 2024-08-19T14:55:01.096 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:01.096 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528366 2024-08-19T14:55:01.096 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:01.096 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T14:55:01.096 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528366 2024-08-19T14:55:01.096 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:01.096 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528366 2024-08-19T14:55:01.099 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.:2211: flush_pg_stats: osd=5 2024-08-19T14:55:01.099 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-481036337154 2024-08-19T14:55:01.099 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:01.099 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=481036337154 2024-08-19T14:55:01.099 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 481036337154' 2024-08-19T14:55:01.099 INFO:tasks.workunit.client.0.smithi102.stderr:223924: waiting osd.5 seq 481036337154 2024-08-19T14:55:01.099 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T14:55:01.099 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 481036337155 -lt 481036337154 2024-08-19T14:55:01.100 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:01.100 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855424 2024-08-19T14:55:01.100 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:01.100 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T14:55:01.100 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855424 2024-08-19T14:55:01.100 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:01.100 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855424 2024-08-19T14:55:01.100 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855424' 2024-08-19T14:55:01.100 INFO:tasks.workunit.client.0.smithi102.stderr:223924: waiting osd.6 seq 150323855424 2024-08-19T14:55:01.100 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T14:55:01.100 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855425 -lt 150323855424 2024-08-19T14:55:01.101 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:01.101 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691898 2024-08-19T14:55:01.101 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:01.101 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T14:55:01.101 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691898 2024-08-19T14:55:01.101 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:01.101 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691898 2024-08-19T14:55:01.101 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691898' 2024-08-19T14:55:01.101 INFO:tasks.workunit.client.0.smithi102.stderr:223924: waiting osd.7 seq 171798691898 2024-08-19T14:55:01.101 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T14:55:01.101 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691899 -lt 171798691898 2024-08-19T14:55:01.101 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:01.101 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528367 2024-08-19T14:55:01.101 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:01.101 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T14:55:01.102 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528367 2024-08-19T14:55:01.102 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:01.102 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528367 2024-08-19T14:55:02.690 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/cloneclient.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528366' 2024-08-19T14:55:02.690 INFO:tasks.workunit.client.0.smithi102.stderr:223922: waiting osd.8 seq 193273528366 2024-08-19T14:55:02.690 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T14:55:02.691 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528368 -lt 193273528366 2024-08-19T14:55:02.691 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:02.691 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566088 2024-08-19T14:55:02.691 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:02.691 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T14:55:02.691 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566088 2024-08-19T14:55:02.691 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:02.691 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566088 2024-08-19T14:55:02.691 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566088' 2024-08-19T14:55:02.691 INFO:tasks.workunit.client.0.smithi102.stderr:223922: waiting osd.9 seq 450971566088 2024-08-19T14:55:02.691 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T14:55:02.691 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566089 -lt 450971566088 2024-08-19T14:55:02.692 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:55:02.692 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:55:02.692 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:55:02.692 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:55:02.692 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:55:02.692 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:55:02.692 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:55:02.692 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:55:02.692 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:55:02.692 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:55:02.692 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:55:02.692 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:55:02.692 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:55:02.693 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:55:02.693 INFO:tasks.workunit.client.0.smithi102.stderr:223922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:55:02.693 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:55:02.693 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:55:02.693 INFO:tasks.workunit.client.0.smithi102.stderr:223922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:55:02.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:55:02.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:55:02.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 223921 2024-08-19T14:55:02.802 INFO:tasks.workunit.client.0.smithi102.stderr:.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528367' 2024-08-19T14:55:02.802 INFO:tasks.workunit.client.0.smithi102.stderr:223924: waiting osd.8 seq 193273528367 2024-08-19T14:55:02.802 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T14:55:02.802 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528368 -lt 193273528367 2024-08-19T14:55:02.802 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:02.802 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566087 2024-08-19T14:55:02.803 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:02.803 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T14:55:02.803 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566087 2024-08-19T14:55:02.803 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:02.803 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566087 2024-08-19T14:55:02.803 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566087' 2024-08-19T14:55:02.803 INFO:tasks.workunit.client.0.smithi102.stderr:223924: waiting osd.9 seq 450971566087 2024-08-19T14:55:02.803 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T14:55:02.803 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566089 -lt 450971566087 2024-08-19T14:55:02.803 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:55:02.803 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:55:02.804 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:55:02.804 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:55:02.804 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:55:02.804 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:55:02.804 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:55:02.804 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:55:02.804 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:55:02.804 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:55:02.804 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:55:02.804 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:55:02.804 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:55:02.804 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:55:02.804 INFO:tasks.workunit.client.0.smithi102.stderr:223924: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:55:02.805 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:55:02.805 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:55:02.805 INFO:tasks.workunit.client.0.smithi102.stderr:223924: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:55:02.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:55:02.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-08-19T14:55:02.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-08-19T14:55:02.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-08-19T14:55:02.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-08-19T14:55:02.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-08-19T14:55:02.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-08-19T14:55:02.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T14:55:02.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T14:55:02.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T14:55:02.806 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T14:55:03.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-08-19T14:55:03.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-08-19T14:55:03.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-08-19T14:55:03.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-08-19T14:55:03.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:55:03.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:55:03.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:55:03.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:55:03.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T14:54:28.504593+0000 2024-08-19T14:55:03.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-08-19T14:55:03.614 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-08-19T14:55:03.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-08-19T14:54:28.504593+0000 2024-08-19T14:55:03.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T14:55:03.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T14:54:28.504593+0000 2024-08-19T14:55:03.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T14:55:03.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T14:55:03.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:55:03.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:55:03.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:55:03.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:55:03.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:55:03.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:55:03.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:54:28.504593+0000 '>' 2024-08-19T14:54:28.504593+0000 2024-08-19T14:55:03.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:55:04.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:55:04.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:55:04.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:55:04.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:55:04.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:55:04.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:55:04.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:55:05.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:54:28.504593+0000 '>' 2024-08-19T14:54:28.504593+0000 2024-08-19T14:55:05.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:55:06.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:55:06.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:55:06.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:55:06.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:55:06.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:55:06.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:55:06.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:55:06.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:54:28.504593+0000 '>' 2024-08-19T14:54:28.504593+0000 2024-08-19T14:55:06.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:55:07.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:55:07.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:55:07.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:55:07.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:55:07.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:55:07.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:55:07.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:55:07.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:54:28.504593+0000 '>' 2024-08-19T14:54:28.504593+0000 2024-08-19T14:55:07.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:55:08.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:55:08.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:55:08.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:55:08.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:55:08.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:55:08.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:55:08.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:55:09.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:54:28.504593+0000 '>' 2024-08-19T14:54:28.504593+0000 2024-08-19T14:55:09.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:55:10.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:55:10.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:55:10.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:55:10.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:55:10.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:55:10.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:55:10.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:55:10.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:54:28.504593+0000 '>' 2024-08-19T14:54:28.504593+0000 2024-08-19T14:55:10.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T14:55:11.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T14:55:11.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T14:55:11.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T14:55:11.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T14:55:11.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T14:55:11.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T14:55:11.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T14:55:11.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T14:55:04.585198+0000 '>' 2024-08-19T14:54:28.504593+0000 2024-08-19T14:55:11.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T14:55:11.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-08-19T14:55:11.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:55:11.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:55:11.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:55:11.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 227421"' 2024-08-19T14:55:11.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 227421' 2024-08-19T14:55:11.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-08-19T14:55:11.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T14:55:11.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T14:55:11.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 227422"' 2024-08-19T14:55:11.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 227422' 2024-08-19T14:55:11.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-08-19T14:55:11.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 227421 227422' 2024-08-19T14:55:11.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-08-19T14:55:11.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:55:11.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 227421 2024-08-19T14:55:11.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:55:11.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/227424: /' 2024-08-19T14:55:11.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-08-19T14:55:11.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/227426: /' 2024-08-19T14:55:14.349 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:55:14.349 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:55:14.349 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-08-19T14:55:14.349 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:55:14.349 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:55:14.349 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:55:14.350 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:55:14.350 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-08-19T14:55:14.350 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:55:14.350 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-08-19T14:55:14.350 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:55:14.350 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:55:14.350 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:55:14.350 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:55:14.350 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:55:14.350 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:55:14.350 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T14:55:14.350 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:55:14.350 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:55:14.351 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-08-19T14:55:14.351 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:55:14.351 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:55:14.351 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-08-19T14:55:14.351 INFO:tasks.workunit.client.0.smithi102.stderr:227424: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-08-19T14:55:14.351 INFO:tasks.workunit.client.0.smithi102.stderr:227424: _ 2024-08-19T14:55:14.351 INFO:tasks.workunit.client.0.smithi102.stderr:227424: hinfo_key 2024-08-19T14:55:14.351 INFO:tasks.workunit.client.0.smithi102.stderr:227424: snapset 2024-08-19T14:55:14.351 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-08-19T14:55:14.351 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:55:14.351 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:55:14.351 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-19T14:55:14.352 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:55:14.352 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:55:14.352 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:55:14.352 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:55:14.352 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:55:14.352 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:55:14.353 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T14:55:14.353 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T14:55:14.353 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:55:14.353 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:55:14.353 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:55:14.353 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:55:14.353 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:55:14.353 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:55:14.353 INFO:tasks.workunit.client.0.smithi102.stderr:227424: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:55:14.353 INFO:tasks.workunit.client.0.smithi102.stderr:227424: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:55:14.353 INFO:tasks.workunit.client.0.smithi102.stderr:227424: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:55:14.353 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:55:14.354 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:55:14.354 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:55:14.354 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:55:14.354 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:55:14.354 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:55:14.354 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:55:14.354 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:55:14.354 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:55:14.354 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:55:14.354 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:55:14.354 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T14:55:14.354 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-19T14:55:14.355 INFO:tasks.workunit.client.0.smithi102.stderr:227424: start osd.3 2024-08-19T14:55:14.652 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T14:55:14.653 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T14:55:14.653 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-08-19T14:55:14.653 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T14:55:14.653 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-08-19T14:55:14.653 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T14:55:14.653 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T14:55:14.653 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-08-19T14:55:14.653 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T14:55:14.653 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-08-19T14:55:14.653 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:55:14.653 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:55:14.654 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:55:14.654 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:55:14.654 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:55:14.654 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:55:14.654 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-08-19T14:55:14.654 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T14:55:14.654 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T14:55:14.654 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-08-19T14:55:14.654 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T14:55:14.654 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-08-19T14:55:14.654 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-08-19T14:55:14.655 INFO:tasks.workunit.client.0.smithi102.stderr:227426: _ 2024-08-19T14:55:14.655 INFO:tasks.workunit.client.0.smithi102.stderr:227426: hinfo_key 2024-08-19T14:55:14.655 INFO:tasks.workunit.client.0.smithi102.stderr:227426: snapset 2024-08-19T14:55:14.655 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-08-19T14:55:14.655 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T14:55:14.655 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T14:55:14.655 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-08-19T14:55:14.655 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T14:55:14.655 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-08-19T14:55:14.655 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:55:14.655 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:55:14.656 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:55:14.656 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:55:14.656 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-08-19T14:55:14.656 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-08-19T14:55:14.656 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T14:55:14.656 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T14:55:14.656 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:55:14.656 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T14:55:14.656 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:55:14.657 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:55:14.657 INFO:tasks.workunit.client.0.smithi102.stderr:227426: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:55:14.657 INFO:tasks.workunit.client.0.smithi102.stderr:227426: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:55:14.657 INFO:tasks.workunit.client.0.smithi102.stderr:227426: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:55:14.657 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:55:14.657 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:55:14.657 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:55:14.657 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:55:14.657 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:55:14.657 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:55:14.657 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:55:14.658 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:55:14.658 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:55:14.658 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T14:55:14.658 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T14:55:14.658 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-08-19T14:55:14.658 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-08-19T14:55:14.658 INFO:tasks.workunit.client.0.smithi102.stderr:227426: start osd.5 2024-08-19T14:55:18.610 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:55:18.610 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-08-19T14:55:18.610 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-19T14:55:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:55:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:55:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:55:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:227424: 2024-08-19T14:55:14.382+0000 7f98bb410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:55:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:227424: 2024-08-19T14:55:14.384+0000 7f98bb410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:55:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:227424: 2024-08-19T14:55:14.386+0000 7f98bb410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:55:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-19T14:55:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:55:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T14:55:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:55:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:55:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:55:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:227424: 0 2024-08-19T14:55:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:55:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:227424: 2024-08-19T14:55:14.900+0000 7f98bb410640 -1 Falling back to public interface 2024-08-19T14:55:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:55:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:227424: 1 2024-08-19T14:55:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:55:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:55:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:227424: 2 2024-08-19T14:55:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:55:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:227424: 2024-08-19T14:55:17.261+0000 7f98bb410640 -1 osd.3 114 log_to_monitors true 2024-08-19T14:55:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:55:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:227424: 3 2024-08-19T14:55:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:19.159 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-08-19T14:55:19.160 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-08-19T14:55:19.160 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-08-19T14:55:19.161 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T14:55:19.161 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T14:55:19.161 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T14:55:19.161 INFO:tasks.workunit.client.0.smithi102.stderr:227426: 2024-08-19T14:55:14.681+0000 7f3ed680d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:55:19.161 INFO:tasks.workunit.client.0.smithi102.stderr:227426: 2024-08-19T14:55:14.727+0000 7f3ed680d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:55:19.161 INFO:tasks.workunit.client.0.smithi102.stderr:227426: 2024-08-19T14:55:14.728+0000 7f3ed680d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:55:19.161 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-08-19T14:55:19.161 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:55:19.161 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-08-19T14:55:19.161 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:55:19.162 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:55:19.162 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:19.162 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:55:19.162 INFO:tasks.workunit.client.0.smithi102.stderr:227426: 0 2024-08-19T14:55:19.162 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:19.163 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:55:19.163 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:19.163 INFO:tasks.workunit.client.0.smithi102.stderr:227426: 2024-08-19T14:55:15.244+0000 7f3ed680d640 -1 Falling back to public interface 2024-08-19T14:55:19.163 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:19.163 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:19.163 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:55:19.163 INFO:tasks.workunit.client.0.smithi102.stderr:227426: 1 2024-08-19T14:55:19.163 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:19.163 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:55:19.163 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:19.163 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:19.164 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:19.164 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:55:19.164 INFO:tasks.workunit.client.0.smithi102.stderr:227426: 2 2024-08-19T14:55:19.164 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:19.164 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:55:19.164 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:19.164 INFO:tasks.workunit.client.0.smithi102.stderr:227426: 2024-08-19T14:55:18.053+0000 7f3ed680d640 -1 osd.5 114 log_to_monitors true 2024-08-19T14:55:19.164 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:19.164 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:19.164 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:55:19.164 INFO:tasks.workunit.client.0.smithi102.stderr:227426: 3 2024-08-19T14:55:19.164 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:19.164 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:55:19.986 INFO:tasks.workunit.client.0.smithi102.stderr:227426: osd.5 up in weight 1 up_from 120 up_thru 112 dowrs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:55:19.986 INFO:tasks.workunit.client.0.smithi102.stderr:227424: osd.3 up in weight 1 up_from 118 up_thru 118 down_at 115 last_clean_interval [112,114) [v2:127.0.0.1:6826/3676545867,v1:127.0.0.1:6827/3676545867] [v2:127.0.0.1:6828/3676545867,v1:127.0.0.1:6829/3676545867] exists,up 0a924bc7-1b05-4420-8781-adc75bdf85fc 2024-08-19T14:55:19.986 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:55:19.986 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:55:19.986 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:55:19.986 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:55:19.986 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:55:19.987 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:55:19.987 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:55:19.987 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:55:19.987 INFO:tasks.workunit.client.0.smithi102.stderr:227424: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:55:19.987 INFO:tasks.workunit.client.0.smithi102.stderr:227424: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:55:19.987 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:55:19.987 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:55:19.987 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:55:19.987 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:55:19.988 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:55:19.988 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:55:19.988 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:55:19.988 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:55:19.988 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:55:19.988 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:55:19.988 INFO:tasks.workunit.client.0.smithi102.stderr:227424: 1 2024-08-19T14:55:19.988 INFO:tasks.workunit.client.0.smithi102.stderr:227424: 2 2024-08-19T14:55:19.988 INFO:tasks.workunit.client.0.smithi102.stderr:227424: 3 2024-08-19T14:55:19.988 INFO:tasks.workunit.client.0.smithi102.stderr:227424: 4 2024-08-19T14:55:19.989 INFO:tasks.workunit.client.0.smithi102.stderr:227424: 5 2024-08-19T14:55:19.989 INFO:tasks.workunit.client.0.smithi102.stderr:227424: 6 2024-08-19T14:55:19.989 INFO:tasks.workunit.client.0.smithi102.stderr:227424: 7 2024-08-19T14:55:19.989 INFO:tasks.workunit.client.0.smithi102.stderr:227424: 8 2024-08-19T14:55:19.989 INFO:tasks.workunit.client.0.smithi102.stderr:227424: 9' 2024-08-19T14:55:19.989 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:55:19.990 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:19.990 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:55:19.990 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836593 2024-08-19T14:55:19.990 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836593 2024-08-19T14:55:19.990 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593' 2024-08-19T14:55:19.990 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:19.990 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:55:19.991 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673062 2024-08-19T14:55:19.991 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673062 2024-08-19T14:55:19.991 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673062' 2024-08-19T14:55:20.268 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0n_at 115 last_clean_interval [112,114) [v2:127.0.0.1:6874/3587193844,v1:127.0.0.1:6875/3587193844] [v2:127.0.0.1:6876/3587193844,v1:127.0.0.1:6877/3587193844] exists,up 7c59bef0-d717-44f2-b3df-88b33605d1af 2024-08-19T14:55:20.268 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:55:20.268 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:55:20.268 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:55:20.268 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T14:55:20.269 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T14:55:20.269 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T14:55:20.269 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T14:55:20.269 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T14:55:20.269 INFO:tasks.workunit.client.0.smithi102.stderr:227426: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T14:55:20.269 INFO:tasks.workunit.client.0.smithi102.stderr:227426: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T14:55:20.269 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T14:55:20.269 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T14:55:20.269 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T14:55:20.269 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T14:55:20.269 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T14:55:20.269 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T14:55:20.269 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T14:55:20.269 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T14:55:20.270 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T14:55:20.270 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T14:55:20.270 INFO:tasks.workunit.client.0.smithi102.stderr:227426: 1 2024-08-19T14:55:20.270 INFO:tasks.workunit.client.0.smithi102.stderr:227426: 2 2024-08-19T14:55:20.270 INFO:tasks.workunit.client.0.smithi102.stderr:227426: 3 2024-08-19T14:55:20.270 INFO:tasks.workunit.client.0.smithi102.stderr:227426: 4 2024-08-19T14:55:20.270 INFO:tasks.workunit.client.0.smithi102.stderr:227426: 5 2024-08-19T14:55:20.270 INFO:tasks.workunit.client.0.smithi102.stderr:227426: 6 2024-08-19T14:55:20.270 INFO:tasks.workunit.client.0.smithi102.stderr:227426: 7 2024-08-19T14:55:20.270 INFO:tasks.workunit.client.0.smithi102.stderr:227426: 8 2024-08-19T14:55:20.270 INFO:tasks.workunit.client.0.smithi102.stderr:227426: 9' 2024-08-19T14:55:20.270 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T14:55:20.270 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T14:55:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836594 2024-08-19T14:55:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836594 2024-08-19T14:55:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594' 2024-08-19T14:55:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T14:55:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673063 2024-08-19T14:55:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673063 2024-08-19T14:55:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673063' 2024-08-19T14:55:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:21.029 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest//qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:21.029 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:55:21.029 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509536 2024-08-19T14:55:21.029 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509536 2024-08-19T14:55:21.029 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673062 2-64424509536' 2024-08-19T14:55:21.030 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:21.030 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:55:21.030 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140930 2024-08-19T14:55:21.030 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140930 2024-08-19T14:55:21.030 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673062 2-64424509536 3-506806140930' 2024-08-19T14:55:21.030 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:21.030 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T14:55:21.030 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182480 2024-08-19T14:55:21.030 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182480 2024-08-19T14:55:21.030 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673062 2-64424509536 3-506806140930 4-107374182480' 2024-08-19T14:55:21.030 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:21.030 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T14:55:21.030 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=515396075522 2024-08-19T14:55:21.031 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 515396075522 2024-08-19T14:55:21.031 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673062 2-64424509536 3-506806140930 4-107374182480 5-515396075522' 2024-08-19T14:55:21.031 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:21.031 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T14:55:21.031 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855430 2024-08-19T14:55:21.031 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855430 2024-08-19T14:55:21.031 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673062 2-64424509536 3-506806140930 4-107374182480 5-515396075522 6-150323855430' 2024-08-19T14:55:21.031 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:21.031 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T14:55:21.031 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691904 2024-08-19T14:55:21.031 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691904 2024-08-19T14:55:21.031 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673062 2-64424509536 3-506806140930 4-107374182480 5-515396075522 6-150323855430 7-171798691904' 2024-08-19T14:55:21.031 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:21.264 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qclone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T14:55:21.264 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509537 2024-08-19T14:55:21.265 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509537 2024-08-19T14:55:21.265 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673063 2-64424509537' 2024-08-19T14:55:21.265 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:21.265 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T14:55:21.265 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140931 2024-08-19T14:55:21.265 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140931 2024-08-19T14:55:21.265 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673063 2-64424509537 3-506806140931' 2024-08-19T14:55:21.265 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:21.265 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T14:55:21.265 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182481 2024-08-19T14:55:21.265 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182481 2024-08-19T14:55:21.265 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673063 2-64424509537 3-506806140931 4-107374182481' 2024-08-19T14:55:21.265 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:21.266 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T14:55:21.266 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=515396075523 2024-08-19T14:55:21.266 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 515396075523 2024-08-19T14:55:21.266 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673063 2-64424509537 3-506806140931 4-107374182481 5-515396075523' 2024-08-19T14:55:21.266 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:21.266 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T14:55:21.266 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855431 2024-08-19T14:55:21.266 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855431 2024-08-19T14:55:21.266 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673063 2-64424509537 3-506806140931 4-107374182481 5-515396075523 6-150323855431' 2024-08-19T14:55:21.266 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:21.266 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T14:55:21.266 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691905 2024-08-19T14:55:21.266 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691905 2024-08-19T14:55:21.267 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673063 2-64424509537 3-506806140931 4-107374182481 5-515396075523 6-150323855431 7-171798691905' 2024-08-19T14:55:21.267 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:21.267 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T14:55:21.854 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuna/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T14:55:21.854 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528373 2024-08-19T14:55:21.854 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528373 2024-08-19T14:55:21.854 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673062 2-64424509536 3-506806140930 4-107374182480 5-515396075522 6-150323855430 7-171798691904 8-193273528373' 2024-08-19T14:55:21.854 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:21.854 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T14:55:21.854 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566094 2024-08-19T14:55:21.855 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566094 2024-08-19T14:55:21.855 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673062 2-64424509536 3-506806140930 4-107374182480 5-515396075522 6-150323855430 7-171798691904 8-193273528373 9-450971566094' 2024-08-19T14:55:21.855 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:21.855 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836593 2024-08-19T14:55:21.855 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:21.855 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:55:21.855 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836593 2024-08-19T14:55:21.855 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:21.855 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836593 2024-08-19T14:55:21.855 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2024-08-19T14:55:21.855 INFO:tasks.workunit.client.0.smithi102.stderr:227424: waiting osd.0 seq 21474836593 2024-08-19T14:55:21.855 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:55:21.856 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836594 -lt 21474836593 2024-08-19T14:55:21.856 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:21.856 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673062 2024-08-19T14:55:21.856 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:21.856 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:55:21.856 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673062 2024-08-19T14:55:21.856 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:21.856 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673062 2024-08-19T14:55:21.856 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673062' 2024-08-19T14:55:21.856 INFO:tasks.workunit.client.0.smithi102.stderr:227424: waiting osd.1 seq 42949673062 2024-08-19T14:55:21.856 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:55:21.857 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673063 -lt 42949673062 2024-08-19T14:55:21.857 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:21.857 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509536 2024-08-19T14:55:21.857 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:21.999 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/ceptu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528374 2024-08-19T14:55:21.999 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528374 2024-08-19T14:55:21.999 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673063 2-64424509537 3-506806140931 4-107374182481 5-515396075523 6-150323855431 7-171798691905 8-193273528374' 2024-08-19T14:55:22.000 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T14:55:22.000 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T14:55:22.000 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566095 2024-08-19T14:55:22.000 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566095 2024-08-19T14:55:22.000 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673063 2-64424509537 3-506806140931 4-107374182481 5-515396075523 6-150323855431 7-171798691905 8-193273528374 9-450971566095' 2024-08-19T14:55:22.000 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:22.000 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836594 2024-08-19T14:55:22.000 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:22.000 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T14:55:22.000 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836594 2024-08-19T14:55:22.000 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:22.000 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836594 2024-08-19T14:55:22.000 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836594' 2024-08-19T14:55:22.001 INFO:tasks.workunit.client.0.smithi102.stderr:227426: waiting osd.0 seq 21474836594 2024-08-19T14:55:22.001 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T14:55:22.001 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836594 -lt 21474836594 2024-08-19T14:55:22.001 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:22.001 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673063 2024-08-19T14:55:22.001 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:22.001 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T14:55:22.001 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673063 2024-08-19T14:55:22.001 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:22.001 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673063 2024-08-19T14:55:22.001 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673063' 2024-08-19T14:55:22.001 INFO:tasks.workunit.client.0.smithi102.stderr:227426: waiting osd.1 seq 42949673063 2024-08-19T14:55:22.001 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T14:55:22.002 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673063 -lt 42949673063 2024-08-19T14:55:22.002 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:22.002 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509537 2024-08-19T14:55:22.002 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:22.002 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:55:22.745 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntuhtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T14:55:22.745 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509536 2024-08-19T14:55:22.745 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:22.745 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509536 2024-08-19T14:55:22.745 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-08-19T14:55:22.745 INFO:tasks.workunit.client.0.smithi102.stderr:227424: waiting osd.2 seq 64424509536 2024-08-19T14:55:22.745 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:55:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509537 -lt 64424509536 2024-08-19T14:55:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-506806140930 2024-08-19T14:55:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:55:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-506806140930 2024-08-19T14:55:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=506806140930 2024-08-19T14:55:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 506806140930' 2024-08-19T14:55:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:227424: waiting osd.3 seq 506806140930 2024-08-19T14:55:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:55:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140930 -lt 506806140930 2024-08-19T14:55:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:22.747 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182480 2024-08-19T14:55:22.747 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:22.747 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T14:55:22.747 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182480 2024-08-19T14:55:22.747 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:22.747 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182480 2024-08-19T14:55:22.747 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182480' 2024-08-19T14:55:22.747 INFO:tasks.workunit.client.0.smithi102.stderr:227424: waiting osd.4 seq 107374182480 2024-08-19T14:55:22.747 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T14:55:22.747 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182481 -lt 107374182480 2024-08-19T14:55:22.747 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:22.747 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-515396075522 2024-08-19T14:55:22.747 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:22.748 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T14:55:22.748 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-515396075522 2024-08-19T14:55:22.748 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:22.937 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=5153/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509537 2024-08-19T14:55:22.938 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:22.938 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509537 2024-08-19T14:55:22.938 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2024-08-19T14:55:22.938 INFO:tasks.workunit.client.0.smithi102.stderr:227426: waiting osd.2 seq 64424509537 2024-08-19T14:55:22.938 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T14:55:22.938 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509537 -lt 64424509537 2024-08-19T14:55:22.938 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:22.938 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-506806140931 2024-08-19T14:55:22.938 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:22.938 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T14:55:22.938 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-506806140931 2024-08-19T14:55:22.939 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:22.939 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=506806140931 2024-08-19T14:55:22.939 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 506806140931' 2024-08-19T14:55:22.939 INFO:tasks.workunit.client.0.smithi102.stderr:227426: waiting osd.3 seq 506806140931 2024-08-19T14:55:22.939 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T14:55:22.939 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140931 -lt 506806140931 2024-08-19T14:55:22.939 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:22.939 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182481 2024-08-19T14:55:22.939 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:22.939 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T14:55:22.939 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182481 2024-08-19T14:55:22.939 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:22.939 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182481 2024-08-19T14:55:22.939 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182481' 2024-08-19T14:55:22.940 INFO:tasks.workunit.client.0.smithi102.stderr:227426: waiting osd.4 seq 107374182481 2024-08-19T14:55:22.940 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T14:55:22.940 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182481 -lt 107374182481 2024-08-19T14:55:22.940 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:22.940 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-515396075523 2024-08-19T14:55:22.940 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:22.940 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T14:55:22.940 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-515396075523 2024-08-19T14:55:22.940 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:22.940 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=515396075523 2024-08-19T14:55:23.961 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_st96075522 2024-08-19T14:55:23.961 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 515396075522' 2024-08-19T14:55:23.961 INFO:tasks.workunit.client.0.smithi102.stderr:227424: waiting osd.5 seq 515396075522 2024-08-19T14:55:23.961 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T14:55:23.961 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 515396075523 -lt 515396075522 2024-08-19T14:55:23.961 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:23.962 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855430 2024-08-19T14:55:23.962 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:23.962 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T14:55:23.962 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855430 2024-08-19T14:55:23.962 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:23.962 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855430 2024-08-19T14:55:23.962 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855430' 2024-08-19T14:55:23.962 INFO:tasks.workunit.client.0.smithi102.stderr:227424: waiting osd.6 seq 150323855430 2024-08-19T14:55:23.962 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T14:55:23.962 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855432 -lt 150323855430 2024-08-19T14:55:23.962 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:23.962 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691904 2024-08-19T14:55:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T14:55:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691904 2024-08-19T14:55:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691904 2024-08-19T14:55:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691904' 2024-08-19T14:55:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:227424: waiting osd.7 seq 171798691904 2024-08-19T14:55:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T14:55:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691905 -lt 171798691904 2024-08-19T14:55:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528373 2024-08-19T14:55:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T14:55:23.964 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528373 2024-08-19T14:55:23.964 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:23.964 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528373 2024-08-19T14:55:23.964 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528373' 2024-08-19T14:55:23.964 INFO:tasks.workunit.client.0.smithi102.stderr:227424: waiting osd.8 seq 193273528373 2024-08-19T14:55:23.964 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T14:55:24.150 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1ats: echo 'waiting osd.5 seq 515396075523' 2024-08-19T14:55:24.150 INFO:tasks.workunit.client.0.smithi102.stderr:227426: waiting osd.5 seq 515396075523 2024-08-19T14:55:24.150 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T14:55:24.150 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 515396075523 -lt 515396075523 2024-08-19T14:55:24.150 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:24.150 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855431 2024-08-19T14:55:24.150 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:24.150 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T14:55:24.150 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855431 2024-08-19T14:55:24.150 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:24.150 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855431 2024-08-19T14:55:24.151 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855431' 2024-08-19T14:55:24.151 INFO:tasks.workunit.client.0.smithi102.stderr:227426: waiting osd.6 seq 150323855431 2024-08-19T14:55:24.151 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T14:55:24.151 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855432 -lt 150323855431 2024-08-19T14:55:24.151 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:24.151 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691905 2024-08-19T14:55:24.151 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:24.151 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T14:55:24.151 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691905 2024-08-19T14:55:24.151 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:24.151 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691905 2024-08-19T14:55:24.151 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691905' 2024-08-19T14:55:24.151 INFO:tasks.workunit.client.0.smithi102.stderr:227426: waiting osd.7 seq 171798691905 2024-08-19T14:55:24.152 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T14:55:24.152 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691905 -lt 171798691905 2024-08-19T14:55:24.152 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:24.152 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528374 2024-08-19T14:55:24.152 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:24.152 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T14:55:24.152 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528374 2024-08-19T14:55:24.152 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:24.152 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528374 2024-08-19T14:55:24.158 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528374' 2024-08-19T14:55:24.158 INFO:tasks.workunit.client.0.smithi102.stderr:227426: waiting osd.8 seq 193273528374 2024-08-19T14:55:24.158 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T14:55:24.158 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528374 -lt 193273528374 2024-08-19T14:55:25.223 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.93273528374 -lt 193273528373 2024-08-19T14:55:25.223 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:25.223 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566094 2024-08-19T14:55:25.223 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:25.223 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T14:55:25.223 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566094 2024-08-19T14:55:25.223 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:25.223 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566094 2024-08-19T14:55:25.224 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566094' 2024-08-19T14:55:25.224 INFO:tasks.workunit.client.0.smithi102.stderr:227424: waiting osd.9 seq 450971566094 2024-08-19T14:55:25.224 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T14:55:25.224 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566095 -lt 450971566094 2024-08-19T14:55:25.224 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:55:25.224 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:55:25.224 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:55:25.224 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:55:25.224 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:55:25.224 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:55:25.224 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:55:25.224 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:55:25.224 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:55:25.224 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:55:25.225 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:55:25.225 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:55:25.225 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:55:25.225 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:55:25.225 INFO:tasks.workunit.client.0.smithi102.stderr:227424: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:55:25.225 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:55:25.225 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:55:25.225 INFO:tasks.workunit.client.0.smithi102.stderr:227424: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:55:25.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:55:25.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T14:55:25.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 227422 2024-08-19T14:55:25.434 INFO:tasks.workunit.client.0.smithi102.stderr:sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T14:55:25.434 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566095 2024-08-19T14:55:25.434 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T14:55:25.435 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T14:55:25.435 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566095 2024-08-19T14:55:25.435 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T14:55:25.435 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566095 2024-08-19T14:55:25.435 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566095' 2024-08-19T14:55:25.435 INFO:tasks.workunit.client.0.smithi102.stderr:227426: waiting osd.9 seq 450971566095 2024-08-19T14:55:25.435 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T14:55:25.435 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566095 -lt 450971566095 2024-08-19T14:55:25.435 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T14:55:25.435 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:55:25.435 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:55:25.435 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T14:55:25.435 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T14:55:25.436 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T14:55:25.436 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T14:55:25.436 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T14:55:25.436 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T14:55:25.436 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T14:55:25.436 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T14:55:25.436 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T14:55:25.436 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T14:55:25.436 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T14:55:25.436 INFO:tasks.workunit.client.0.smithi102.stderr:227426: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T14:55:25.436 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T14:55:25.436 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T14:55:25.436 INFO:tasks.workunit.client.0.smithi102.stderr:227426: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T14:55:25.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T14:55:25.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-08-19T14:55:25.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-08-19T14:55:25.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-08-19T14:55:25.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-08-19T14:55:25.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-08-19T14:55:25.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T14:55:25.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T14:55:25.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T14:55:25.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:55:25.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:55:25.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:55:25.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:55:25.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:55:25.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:55:25.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:55:25.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:55:25.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:55:25.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:55:25.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:55:25.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:55:25.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:55:25.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:55:25.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:55:25.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:55:25.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:55:25.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:55:25.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:55:25.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:55:25.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:55:25.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:55:25.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:55:25.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:55:25.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:55:25.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:55:25.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:55:25.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:55:25.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T14:55:25.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T14:55:25.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T14:55:25.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T14:55:25.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T14:55:25.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T14:55:25.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T14:55:25.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T14:55:25.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T14:55:25.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T14:55:25.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T14:55:25.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T14:55:25.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T14:55:25.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T14:55:25.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T14:55:25.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T14:55:25.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T14:55:25.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T14:55:25.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T14:55:25.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:55:25.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T14:55:25.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T14:55:25.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:55:25.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T14:55:25.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T14:55:25.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T14:55:25.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T14:55:25.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:55:25.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:55:25.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T14:55:25.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T14:55:25.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T14:55:25.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T14:55:25.678 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T14:55:25.678 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:55:25.678 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:55:25.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T14:55:25.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T14:55:25.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T14:55:25.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T14:55:25.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T14:55:25.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2024-08-19T14:55:25.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2024-08-19T14:55:25.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2024-08-19T14:55:25.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-08-19T14:55:25.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=true 2024-08-19T14:55:25.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-08-19T14:55:25.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-08-19T14:55:25.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T14:55:25.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T14:55:25.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T14:55:25.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T14:55:25.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T14:55:25.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-08-19T14:55:26.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T14:55:26.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:55:26.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:55:26.054 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:55:26.054 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:55:26.054 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:55:26.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:55:26.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T14:55:26.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T14:55:26.238 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T14:55:26.238 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:55:26.238 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:55:26.238 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T14:55:26.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T14:55:26.239 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:55:26.239 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:55:26.239 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:55:26.239 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:55:26.239 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:55:26.239 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:55:26.240 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:55:26.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:55:26.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T14:55:26.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T14:55:26.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T14:55:26.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T14:55:26.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T14:55:26.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T14:55:26.318 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T14:55:26.318 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T14:55:26.319 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T14:55:26.319 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T14:55:26.319 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:55:26.319 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:55:26.319 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T14:55:26.319 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T14:55:26.319 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T14:55:26.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-08-19T14:55:26.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T14:55:26.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T14:55:26.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T14:55:26.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T14:55:26.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T14:55:26.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T14:55:27.197 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T14:55:27.197 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:55:27.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:55:27.198 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:55:27.198 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:55:27.198 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:55:27.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:55:27.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:55:27.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T14:55:27.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-08-19T14:55:27.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T14:55:27.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-08-19T14:55:27.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:55:27.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:55:27.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T14:55:27.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:55:27.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T14:55:27.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:55:27.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:55:27.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:55:27.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:55:27.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T14:55:27.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T14:55:27.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:55:27.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:55:27.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:55:27.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:55:27.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:55:27.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:55:27.232 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:55:27.232 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:55:27.232 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:55:27.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:55:27.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:55:27.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:55:27.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:55:27.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:55:27.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:55:27.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:55:27.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:55:27.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:55:27.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:55:27.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:55:27.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:55:27.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:55:27.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T14:55:27.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:55:27.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ac755142-7853-47b3-8efc-f3e05572e198 2024-08-19T14:55:27.234 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 ac755142-7853-47b3-8efc-f3e05572e198 2024-08-19T14:55:27.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ac755142-7853-47b3-8efc-f3e05572e198' 2024-08-19T14:55:27.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:55:27.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDfXMNmLXDaDhAAzOM0LuhYIIXaAiA53kRPeA== 2024-08-19T14:55:27.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDfXMNmLXDaDhAAzOM0LuhYIIXaAiA53kRPeA=="}' 2024-08-19T14:55:27.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ac755142-7853-47b3-8efc-f3e05572e198 -i td/osd-scrub-repair/0/new.json 2024-08-19T14:55:27.454 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:55:27.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T14:55:27.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDfXMNmLXDaDhAAzOM0LuhYIIXaAiA53kRPeA== --osd-uuid ac755142-7853-47b3-8efc-f3e05572e198 2024-08-19T14:55:27.493 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:55:27.492+0000 7f983f901640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:55:27.505 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:55:27.504+0000 7f983f901640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:55:27.506 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:55:27.505+0000 7f983f901640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:55:27.506 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:55:27.505+0000 7f983f901640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:55:27.506 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:55:27.506+0000 7f983f901640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:55:27.506 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:55:27.506+0000 7f983f901640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T14:55:27.597 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:55:27.596+0000 7f983f901640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T14:55:33.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T14:55:33.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:55:33.088 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T14:55:33.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T14:55:33.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:55:33.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T14:55:33.435 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T14:55:33.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:55:33.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:55:33.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:55:33.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:55:33.463 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:55:33.462+0000 7f633dc13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:55:33.465 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:55:33.464+0000 7f633dc13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:55:33.466 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:55:33.465+0000 7f633dc13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:55:33.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T14:55:33.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:55:33.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T14:55:33.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:55:33.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:55:33.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:33.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:55:33.703 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:55:33.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:33.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:55:33.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:33.995 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:55:33.995+0000 7f633dc13640 -1 Falling back to public interface 2024-08-19T14:55:34.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:34.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:34.969 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:55:34.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:55:34.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:34.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:55:35.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:36.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:36.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:36.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:55:36.241 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:55:36.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:36.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:55:36.331 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:55:36.330+0000 7f633dc13640 -1 osd.0 0 log_to_monitors true 2024-08-19T14:55:36.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:37.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:37.516 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:55:37.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:37.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:55:37.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:37.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:55:37.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:38.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:38.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:38.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:55:38.791 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:55:38.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:38.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:55:39.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:40.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:40.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:40.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:55:40.076 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:55:40.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:40.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:55:40.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:41.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:41.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:41.351 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:55:41.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:55:41.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:41.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:55:41.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:42.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:42.630 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:55:42.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:42.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:55:42.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:42.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:55:42.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:43.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:43.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:43.915 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:55:43.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:55:43.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:43.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:55:44.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:45.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:45.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:45.285 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:55:45.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:55:45.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:45.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:55:45.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:46.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:46.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:46.562 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:55:46.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:55:46.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:46.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:55:46.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:47.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:47.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:47.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:55:47.880 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:55:47.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:47.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:55:48.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:49.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:49.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:49.465 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:55:49.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:55:49.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:49.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:55:50.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:51.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:51.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:51.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:55:51.046 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:55:51.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:51.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:55:51.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:52.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:52.840 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:55:52.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:52.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:55:52.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:52.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:55:53.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:54.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:54.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:54.317 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:55:54.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:55:54.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:54.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:55:55.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:56.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:56.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:56.037 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:55:56.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:55:56.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:56.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:55:56.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:57.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:57.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:57.381 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:55:57.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:55:57.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:57.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:55:57.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:55:58.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:55:58.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:55:58.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:55:58.975 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:55:58.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:55:58.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T14:55:59.274 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/663349700,v1:127.0.0.1:6803/663349700] [v2:127.0.0.1:6804/663349700,v1:127.0.0.1:6805/663349700] exists,up ac755142-7853-47b3-8efc-f3e05572e198 2024-08-19T14:55:59.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:55:59.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:55:59.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:55:59.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T14:55:59.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-08-19T14:55:59.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:55:59.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:55:59.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T14:55:59.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:55:59.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T14:55:59.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:55:59.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:55:59.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:55:59.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:55:59.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T14:55:59.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T14:55:59.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:55:59.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:55:59.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:55:59.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:55:59.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:55:59.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:55:59.280 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:55:59.280 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:55:59.280 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:55:59.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:55:59.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:55:59.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:55:59.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:55:59.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:55:59.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:55:59.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:55:59.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:55:59.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:55:59.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:55:59.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:55:59.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:55:59.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:55:59.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T14:55:59.281 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:55:59.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cd545f16-9211-442d-9173-bdcaae59bda6 2024-08-19T14:55:59.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 cd545f16-9211-442d-9173-bdcaae59bda6' 2024-08-19T14:55:59.282 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 cd545f16-9211-442d-9173-bdcaae59bda6 2024-08-19T14:55:59.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:55:59.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD/XMNmNZulERAAGbbxryb6w3ri7nhhMO2zaw== 2024-08-19T14:55:59.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD/XMNmNZulERAAGbbxryb6w3ri7nhhMO2zaw=="}' 2024-08-19T14:55:59.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cd545f16-9211-442d-9173-bdcaae59bda6 -i td/osd-scrub-repair/1/new.json 2024-08-19T14:55:59.661 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:55:59.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T14:55:59.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD/XMNmNZulERAAGbbxryb6w3ri7nhhMO2zaw== --osd-uuid cd545f16-9211-442d-9173-bdcaae59bda6 2024-08-19T14:55:59.702 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:55:59.701+0000 7f3c6fa18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:55:59.745 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:55:59.744+0000 7f3c6fa18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:55:59.746 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:55:59.745+0000 7f3c6fa18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:55:59.747 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:55:59.746+0000 7f3c6fa18640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:55:59.747 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:55:59.746+0000 7f3c6fa18640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:55:59.747 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:55:59.746+0000 7f3c6fa18640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T14:55:59.820 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:55:59.819+0000 7f3c6fa18640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T14:56:05.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T14:56:05.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:56:05.225 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T14:56:05.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T14:56:05.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:56:05.568 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T14:56:05.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T14:56:05.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:56:05.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:56:05.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:56:05.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:56:05.595 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:56:05.594+0000 7f0217e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:56:05.597 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:56:05.596+0000 7f0217e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:56:05.598 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:56:05.597+0000 7f0217e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:56:05.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T14:56:05.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:56:05.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T14:56:05.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:56:05.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:56:05.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:05.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:56:05.839 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:56:05.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:05.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:56:06.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:06.123 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:56:06.123+0000 7f0217e0d640 -1 Falling back to public interface 2024-08-19T14:56:07.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:07.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:07.100 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:56:07.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:56:07.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:07.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:56:07.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:08.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:08.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:08.425 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:56:08.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:56:08.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:08.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:56:08.469 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:56:08.468+0000 7f0217e0d640 -1 osd.1 0 log_to_monitors true 2024-08-19T14:56:08.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:09.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:09.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:09.702 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:56:09.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:56:09.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:09.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:56:10.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:11.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:11.086 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:56:11.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:11.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:56:11.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:11.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:56:11.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:12.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:12.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:12.525 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:56:12.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:56:12.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:12.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:56:12.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:13.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:13.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:13.799 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:56:13.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:56:13.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:13.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:56:14.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:15.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:15.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:15.086 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:56:15.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:56:15.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:15.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:56:15.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:16.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:16.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:16.374 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:56:16.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:56:16.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:16.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:56:16.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:17.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:17.651 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:56:17.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:17.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:56:17.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:17.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:56:18.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:19.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:19.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:19.001 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:56:19.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:56:19.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:19.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:56:19.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:20.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:20.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:20.367 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:56:20.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:56:20.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:20.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:56:20.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:21.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:21.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:21.865 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:56:21.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:56:21.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:21.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:56:22.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:23.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:23.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:23.555 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:56:23.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:56:23.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:23.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:56:24.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:25.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:25.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:25.533 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:56:25.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:56:25.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:25.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:56:26.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:27.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:27.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:27.361 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:56:27.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:56:27.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:27.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:56:28.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:29.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:29.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:29.114 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:56:29.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:56:29.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:29.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:56:29.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:30.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:30.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:30.508 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:56:30.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:56:30.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:30.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:56:30.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:31.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:31.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:31.811 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:56:31.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:56:31.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:31.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T14:56:32.078 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2480406889,v1:127.0.0.1:6811/2480406889] [v2:127.0.0.1:6812/2480406889,v1:127.0.0.1:6813/2480406889] exists,up cd545f16-9211-442d-9173-bdcaae59bda6 2024-08-19T14:56:32.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:56:32.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:56:32.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:56:32.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T14:56:32.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-08-19T14:56:32.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:56:32.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:56:32.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T14:56:32.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:56:32.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T14:56:32.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:56:32.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:56:32.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:56:32.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:56:32.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T14:56:32.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T14:56:32.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:56:32.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:56:32.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:56:32.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:56:32.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:56:32.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:56:32.084 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:56:32.084 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:56:32.085 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:56:32.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:56:32.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:56:32.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:56:32.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:56:32.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:56:32.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:56:32.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:56:32.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:56:32.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:56:32.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:56:32.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:56:32.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:56:32.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:56:32.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T14:56:32.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:56:32.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ee1f08eb-903f-4823-9c48-dde4e2b1f107 2024-08-19T14:56:32.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ee1f08eb-903f-4823-9c48-dde4e2b1f107' 2024-08-19T14:56:32.087 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 ee1f08eb-903f-4823-9c48-dde4e2b1f107 2024-08-19T14:56:32.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:56:32.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAgXcNmS633BRAAaorVMXeRF7CQTL6/xZ85/Q== 2024-08-19T14:56:32.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAgXcNmS633BRAAaorVMXeRF7CQTL6/xZ85/Q=="}' 2024-08-19T14:56:32.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ee1f08eb-903f-4823-9c48-dde4e2b1f107 -i td/osd-scrub-repair/2/new.json 2024-08-19T14:56:32.437 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:56:32.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-08-19T14:56:32.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAgXcNmS633BRAAaorVMXeRF7CQTL6/xZ85/Q== --osd-uuid ee1f08eb-903f-4823-9c48-dde4e2b1f107 2024-08-19T14:56:32.478 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:56:32.476+0000 7f321db01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:56:32.513 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:56:32.512+0000 7f321db01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:56:32.514 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:56:32.513+0000 7f321db01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:56:32.515 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:56:32.513+0000 7f321db01640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:56:32.515 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:56:32.513+0000 7f321db01640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:56:32.515 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:56:32.513+0000 7f321db01640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T14:56:32.588 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:56:32.587+0000 7f321db01640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-08-19T14:56:38.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-08-19T14:56:38.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:56:38.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T14:56:38.407 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T14:56:38.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:56:38.753 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T14:56:38.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T14:56:38.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:56:38.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:56:38.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:56:38.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:56:38.780 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:56:38.780+0000 7f88fbf1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:56:38.782 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:56:38.782+0000 7f88fbf1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:56:38.784 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:56:38.783+0000 7f88fbf1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:56:39.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T14:56:39.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:56:39.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T14:56:39.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:56:39.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:56:39.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:39.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:56:39.024 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:56:39.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:39.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:56:39.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:39.305 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:56:39.304+0000 7f88fbf1a640 -1 Falling back to public interface 2024-08-19T14:56:40.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:40.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:40.292 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:56:40.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:56:40.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:40.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:56:40.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:41.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:41.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:41.564 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:56:41.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:56:41.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:41.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:56:41.774 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:56:41.773+0000 7f88fbf1a640 -1 osd.2 0 log_to_monitors true 2024-08-19T14:56:41.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:42.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:42.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:42.891 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:56:42.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:56:42.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:42.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:56:43.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:44.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:44.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:44.167 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:56:44.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:56:44.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:44.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:56:44.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:45.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:45.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:45.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:56:45.447 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:56:45.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:45.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:56:45.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:46.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:46.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:46.804 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:56:46.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:56:46.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:46.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:56:47.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:48.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:48.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:48.081 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:56:48.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:56:48.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:48.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:56:48.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:49.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:49.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:49.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:56:49.493 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:56:49.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:49.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:56:49.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:50.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:50.784 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:56:50.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:50.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:56:50.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:50.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:56:51.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:52.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:52.059 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:56:52.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:52.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:56:52.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:52.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:56:52.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:53.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:53.334 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:56:53.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:53.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:56:53.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:53.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:56:53.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:54.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:54.611 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:56:54.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:54.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:56:54.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:54.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:56:55.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:56.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:56.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:56.208 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:56:56.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:56:56.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:56.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:56:56.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:57.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:57.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:57.818 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:56:57.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:56:57.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:57.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:56:58.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:56:59.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:56:59.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:56:59.109 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:56:59.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:56:59.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:56:59.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:56:59.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:00.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:00.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:00.537 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:57:00.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:57:00.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:00.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:57:01.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:02.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:02.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:02.140 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:57:02.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:57:02.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:02.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:57:02.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:03.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:03.893 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:57:03.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:03.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:57:03.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:03.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:57:04.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:05.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:05.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:05.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T14:57:05.322 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T14:57:05.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:05.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T14:57:05.619 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2932186873,v1:127.0.0.1:6819/2932186873] [v2:127.0.0.1:6820/2932186873,v1:127.0.0.1:6821/2932186873] exists,up ee1f08eb-903f-4823-9c48-dde4e2b1f107 2024-08-19T14:57:05.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:57:05.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:57:05.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:57:05.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T14:57:05.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-08-19T14:57:05.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:57:05.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:57:05.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-08-19T14:57:05.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:57:05.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T14:57:05.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:57:05.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:57:05.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:57:05.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:57:05.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T14:57:05.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T14:57:05.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:57:05.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:57:05.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:57:05.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:57:05.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:57:05.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:57:05.626 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:57:05.626 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:57:05.626 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:57:05.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:57:05.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:57:05.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:57:05.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:57:05.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:57:05.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:57:05.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:57:05.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:57:05.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:57:05.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:57:05.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:57:05.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:57:05.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:57:05.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T14:57:05.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:57:05.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a53d7b50-dd2b-44ae-862b-772fc9028cb8 2024-08-19T14:57:05.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 a53d7b50-dd2b-44ae-862b-772fc9028cb8' 2024-08-19T14:57:05.628 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 a53d7b50-dd2b-44ae-862b-772fc9028cb8 2024-08-19T14:57:05.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:57:05.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBBXcNmKaMyJhAAkIqQsQ0vSwe5FvZfDAT12A== 2024-08-19T14:57:05.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBBXcNmKaMyJhAAkIqQsQ0vSwe5FvZfDAT12A=="}' 2024-08-19T14:57:05.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a53d7b50-dd2b-44ae-862b-772fc9028cb8 -i td/osd-scrub-repair/3/new.json 2024-08-19T14:57:05.989 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:57:06.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-08-19T14:57:06.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBXcNmKaMyJhAAkIqQsQ0vSwe5FvZfDAT12A== --osd-uuid a53d7b50-dd2b-44ae-862b-772fc9028cb8 2024-08-19T14:57:06.030 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:06.029+0000 7fc1c9ce1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:57:06.074 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:06.073+0000 7fc1c9ce1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:57:06.076 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:06.074+0000 7fc1c9ce1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:57:06.076 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:06.074+0000 7fc1c9ce1640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-08-19T14:57:06.076 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:06.074+0000 7fc1c9ce1640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-08-19T14:57:06.076 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:06.074+0000 7fc1c9ce1640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-08-19T14:57:06.154 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:06.153+0000 7fc1c9ce1640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-08-19T14:57:12.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-08-19T14:57:12.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:57:12.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-08-19T14:57:12.242 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-08-19T14:57:12.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:57:12.586 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-08-19T14:57:12.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-08-19T14:57:12.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:57:12.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:57:12.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:57:12.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:57:12.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:12.612+0000 7fbffe7a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:57:12.614 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:12.614+0000 7fbffe7a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:57:12.616 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:12.615+0000 7fbffe7a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:57:12.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-08-19T14:57:12.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:57:12.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T14:57:12.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:57:12.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:57:12.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:12.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:57:12.855 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:57:12.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:12.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:13.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:13.136 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:13.136+0000 7fbffe7a3640 -1 Falling back to public interface 2024-08-19T14:57:14.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:14.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:14.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:57:14.122 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:57:14.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:14.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:14.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:15.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:15.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:15.398 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:57:15.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:57:15.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:15.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:15.625 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:15.624+0000 7fbffe7a3640 -1 osd.3 0 log_to_monitors true 2024-08-19T14:57:15.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:16.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:16.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:16.678 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:57:16.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:57:16.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:16.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:16.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:17.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:17.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:17.950 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:57:17.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:57:17.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:17.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:18.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:19.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:19.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:19.235 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:57:19.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:57:19.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:19.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:19.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:20.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:20.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:20.716 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:57:20.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:57:20.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:20.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:20.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:21.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:21.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:21.994 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:57:21.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:57:21.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:21.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:22.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:23.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:23.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:23.270 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:57:23.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:57:23.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:23.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:23.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:24.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:24.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:24.725 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:57:24.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:57:24.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:24.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:25.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:26.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:26.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:26.003 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:57:26.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:57:26.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:26.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:26.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:27.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:27.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:27.285 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:57:27.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:57:27.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:27.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:27.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:28.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:28.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:28.873 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:57:28.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:57:28.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:28.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:29.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:30.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:30.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:30.355 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:57:30.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:57:30.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:30.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:30.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:31.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:31.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:31.765 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:57:31.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:57:31.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:31.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:32.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:33.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:33.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:33.388 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:57:33.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:57:33.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:33.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:33.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:34.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:34.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:34.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:57:34.781 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:57:34.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:34.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:35.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:36.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:36.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:36.301 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:57:36.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:57:36.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:36.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:36.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:37.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:37.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:37.801 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:57:37.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:57:37.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:37.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:38.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:39.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:39.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:39.129 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T14:57:39.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T14:57:39.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:39.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T14:57:39.483 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3308022899,v1:127.0.0.1:6827/3308022899] [v2:127.0.0.1:6828/3308022899,v1:127.0.0.1:6829/3308022899] exists,up a53d7b50-dd2b-44ae-862b-772fc9028cb8 2024-08-19T14:57:39.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:57:39.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:57:39.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:57:39.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T14:57:39.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-08-19T14:57:39.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:57:39.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:57:39.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-08-19T14:57:39.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:57:39.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-08-19T14:57:39.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:57:39.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:57:39.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:57:39.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:57:39.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-08-19T14:57:39.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-08-19T14:57:39.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:57:39.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:57:39.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:57:39.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:57:39.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:57:39.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:57:39.489 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:57:39.489 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:57:39.489 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:57:39.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:57:39.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:57:39.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:57:39.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:57:39.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:57:39.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:57:39.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:57:39.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:57:39.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:57:39.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:57:39.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:57:39.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:57:39.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:57:39.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-08-19T14:57:39.490 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:57:39.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9303e3d9-0fbf-4bc7-9cee-c90ff31f8620 2024-08-19T14:57:39.491 INFO:tasks.workunit.client.0.smithi102.stdout:add osd4 9303e3d9-0fbf-4bc7-9cee-c90ff31f8620 2024-08-19T14:57:39.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 9303e3d9-0fbf-4bc7-9cee-c90ff31f8620' 2024-08-19T14:57:39.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:57:39.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBjXcNm8+oQHhAAa9Kqt4C0ErZHIhW9OUTMyA== 2024-08-19T14:57:39.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBjXcNm8+oQHhAAa9Kqt4C0ErZHIhW9OUTMyA=="}' 2024-08-19T14:57:39.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9303e3d9-0fbf-4bc7-9cee-c90ff31f8620 -i td/osd-scrub-repair/4/new.json 2024-08-19T14:57:40.018 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:57:40.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-08-19T14:57:40.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBjXcNm8+oQHhAAa9Kqt4C0ErZHIhW9OUTMyA== --osd-uuid 9303e3d9-0fbf-4bc7-9cee-c90ff31f8620 2024-08-19T14:57:40.059 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:40.058+0000 7ff53460e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:57:40.102 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:40.102+0000 7ff53460e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:57:40.103 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:40.103+0000 7ff53460e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:57:40.104 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:40.103+0000 7ff53460e640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-08-19T14:57:40.104 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:40.103+0000 7ff53460e640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-08-19T14:57:40.104 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:40.103+0000 7ff53460e640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-08-19T14:57:40.177 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:40.176+0000 7ff53460e640 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-08-19T14:57:46.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-08-19T14:57:46.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:57:46.231 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd4 key to auth repository 2024-08-19T14:57:46.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-08-19T14:57:46.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:57:46.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-08-19T14:57:46.574 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.4 2024-08-19T14:57:46.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:57:46.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:57:46.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:57:46.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:57:46.601 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:46.600+0000 7f92de411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:57:46.602 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:46.602+0000 7f92de411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:57:46.603 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:46.603+0000 7f92de411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:57:46.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-08-19T14:57:46.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:57:46.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-08-19T14:57:46.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:57:46.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:57:46.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:46.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:57:46.842 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:57:46.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:46.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:57:47.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:47.125 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:47.125+0000 7f92de411640 -1 Falling back to public interface 2024-08-19T14:57:48.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:48.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:48.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:57:48.108 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:57:48.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:48.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:57:48.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:49.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:49.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:49.381 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:57:49.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:57:49.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:49.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:57:49.736 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:57:49.735+0000 7f92de411640 -1 osd.4 0 log_to_monitors true 2024-08-19T14:57:49.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:50.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:50.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:50.838 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:57:50.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:57:50.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:50.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:57:51.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:52.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:52.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:52.113 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:57:52.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:57:52.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:52.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:57:52.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:53.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:53.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:53.391 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:57:53.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:57:53.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:53.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:57:53.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:54.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:54.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:54.668 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:57:54.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:57:54.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:54.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:57:54.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:55.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:55.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:55.944 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:57:55.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:57:55.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:55.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:57:56.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:57.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:57.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:57.221 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:57:57.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:57:57.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:57.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:57:57.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:58.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:58.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:58.597 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:57:58.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:57:58.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:58.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:57:58.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:57:59.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:57:59.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:57:59.874 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:57:59.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:57:59.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:57:59.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:58:00.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:01.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:01.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:01.151 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:58:01.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:58:01.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:01.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:58:01.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:02.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:02.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:02.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:58:02.493 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:58:02.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:02.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:58:03.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:04.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:04.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:04.041 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:58:04.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:58:04.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:04.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:58:04.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:05.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:05.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:05.410 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:58:05.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:58:05.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:05.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:58:06.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:07.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:07.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:07.027 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:58:07.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:58:07.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:07.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:58:07.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:08.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:08.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:08.725 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:58:08.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:58:08.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:08.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:58:09.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:10.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:10.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:10.379 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:58:10.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:58:10.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:10.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:58:11.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:12.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:12.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:12.107 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:58:12.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:58:12.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:12.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:58:12.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:13.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:13.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:13.533 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T14:58:13.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T14:58:13.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:13.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-08-19T14:58:13.806 INFO:tasks.workunit.client.0.smithi102.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2116648146,v1:127.0.0.1:6835/2116648146] [v2:127.0.0.1:6836/2116648146,v1:127.0.0.1:6837/2116648146] exists,up 9303e3d9-0fbf-4bc7-9cee-c90ff31f8620 2024-08-19T14:58:13.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:58:13.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:58:13.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:58:13.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T14:58:13.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-08-19T14:58:13.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:58:13.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:58:13.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-08-19T14:58:13.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:58:13.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-08-19T14:58:13.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:58:13.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:58:13.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:58:13.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:58:13.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-08-19T14:58:13.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-08-19T14:58:13.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:58:13.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:58:13.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:58:13.813 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:58:13.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:58:13.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:58:13.814 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:58:13.814 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:58:13.814 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:58:13.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:58:13.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:58:13.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:58:13.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:58:13.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:58:13.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:58:13.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:58:13.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:58:13.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:58:13.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:58:13.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:58:13.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:58:13.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:58:13.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-08-19T14:58:13.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:58:13.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=93853ae7-d7eb-460d-96c3-c71aa1567468 2024-08-19T14:58:13.816 INFO:tasks.workunit.client.0.smithi102.stdout:add osd5 93853ae7-d7eb-460d-96c3-c71aa1567468 2024-08-19T14:58:13.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 93853ae7-d7eb-460d-96c3-c71aa1567468' 2024-08-19T14:58:13.816 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:58:13.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCFXcNm+mFcMRAAkfi8DZImMMAEolwmSQwilA== 2024-08-19T14:58:13.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCFXcNm+mFcMRAAkfi8DZImMMAEolwmSQwilA=="}' 2024-08-19T14:58:13.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 93853ae7-d7eb-460d-96c3-c71aa1567468 -i td/osd-scrub-repair/5/new.json 2024-08-19T14:58:14.144 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:58:14.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-08-19T14:58:14.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCFXcNm+mFcMRAAkfi8DZImMMAEolwmSQwilA== --osd-uuid 93853ae7-d7eb-460d-96c3-c71aa1567468 2024-08-19T14:58:14.186 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:14.185+0000 7fa4938dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:58:14.220 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:14.220+0000 7fa4938dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:58:14.221 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:14.221+0000 7fa4938dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:58:14.222 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:14.221+0000 7fa4938dd640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-08-19T14:58:14.222 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:14.221+0000 7fa4938dd640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-08-19T14:58:14.222 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:14.221+0000 7fa4938dd640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-08-19T14:58:14.295 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:14.295+0000 7fa4938dd640 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-08-19T14:58:20.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-08-19T14:58:20.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:58:20.350 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd5 key to auth repository 2024-08-19T14:58:20.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-08-19T14:58:20.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:58:20.694 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.5 2024-08-19T14:58:20.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-08-19T14:58:20.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:58:20.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:58:20.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:58:20.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:58:20.721 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:20.721+0000 7f42de486640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:58:20.723 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:20.723+0000 7f42de486640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:58:20.724 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:20.724+0000 7f42de486640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:58:20.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-08-19T14:58:20.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:58:20.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-08-19T14:58:20.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:58:20.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:58:20.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:20.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:58:20.963 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:58:20.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:20.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:21.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:21.249 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:21.248+0000 7f42de486640 -1 Falling back to public interface 2024-08-19T14:58:22.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:22.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:22.233 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:58:22.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:58:22.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:22.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:22.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:23.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:23.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:23.506 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:58:23.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:58:23.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:23.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:23.713 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:23.713+0000 7f42de486640 -1 osd.5 0 log_to_monitors true 2024-08-19T14:58:23.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:24.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:24.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:24.815 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:58:24.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:58:24.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:24.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:25.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:26.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:26.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:26.092 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:58:26.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:58:26.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:26.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:26.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:27.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:27.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:27.374 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:58:27.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:58:27.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:27.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:27.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:28.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:28.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:28.650 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:58:28.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:58:28.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:28.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:28.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:29.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:29.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:29.925 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:58:29.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:58:29.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:29.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:30.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:31.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:31.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:31.201 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:58:31.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:58:31.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:31.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:31.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:32.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:32.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:32.477 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:58:32.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:58:32.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:32.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:32.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:33.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:33.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:33.754 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:58:33.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:58:33.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:33.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:34.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:35.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:35.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:35.033 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:58:35.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:58:35.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:35.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:35.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:36.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:36.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:36.312 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:58:36.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:58:36.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:36.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:36.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:37.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:37.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:37.712 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:58:37.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:58:37.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:37.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:38.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:39.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:39.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:39.206 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:58:39.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:58:39.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:39.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:39.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:40.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:40.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:40.600 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:58:40.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:58:40.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:40.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:41.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:42.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:42.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:42.216 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:58:42.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:58:42.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:42.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:42.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:43.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:43.850 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:58:43.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:43.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:58:43.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:43.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:44.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:45.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:45.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:45.611 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:58:45.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:58:45.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:45.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:46.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:47.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:47.046 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T14:58:47.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:47.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T14:58:47.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:47.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T14:58:47.322 INFO:tasks.workunit.client.0.smithi102.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2289761390,v1:127.0.0.1:6843/2289761390] [v2:127.0.0.1:6844/2289761390,v1:127.0.0.1:6845/2289761390] exists,up 93853ae7-d7eb-460d-96c3-c71aa1567468 2024-08-19T14:58:47.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:58:47.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:58:47.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:58:47.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T14:58:47.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-08-19T14:58:47.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:58:47.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:58:47.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-08-19T14:58:47.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:58:47.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-08-19T14:58:47.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:58:47.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:58:47.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:58:47.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:58:47.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-08-19T14:58:47.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-08-19T14:58:47.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:58:47.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:58:47.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:58:47.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:58:47.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:58:47.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:58:47.329 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:58:47.329 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:58:47.329 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:58:47.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:58:47.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:58:47.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:58:47.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:58:47.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:58:47.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:58:47.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:58:47.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:58:47.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:58:47.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:58:47.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:58:47.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:58:47.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:58:47.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-08-19T14:58:47.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:58:47.331 INFO:tasks.workunit.client.0.smithi102.stdout:add osd6 08875c1d-be3a-41b7-9e73-295ef385a677 2024-08-19T14:58:47.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=08875c1d-be3a-41b7-9e73-295ef385a677 2024-08-19T14:58:47.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 08875c1d-be3a-41b7-9e73-295ef385a677' 2024-08-19T14:58:47.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:58:47.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCnXcNmw9Z2FBAA8D1fnyKUyxvMKrLN8FVIIQ== 2024-08-19T14:58:47.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCnXcNmw9Z2FBAA8D1fnyKUyxvMKrLN8FVIIQ=="}' 2024-08-19T14:58:47.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 08875c1d-be3a-41b7-9e73-295ef385a677 -i td/osd-scrub-repair/6/new.json 2024-08-19T14:58:47.699 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:58:47.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-08-19T14:58:47.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCnXcNmw9Z2FBAA8D1fnyKUyxvMKrLN8FVIIQ== --osd-uuid 08875c1d-be3a-41b7-9e73-295ef385a677 2024-08-19T14:58:47.739 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:47.738+0000 7f5d3e742640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:58:47.767 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:47.766+0000 7f5d3e742640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:58:47.768 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:47.767+0000 7f5d3e742640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:58:47.769 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:47.768+0000 7f5d3e742640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-08-19T14:58:47.769 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:47.768+0000 7f5d3e742640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-08-19T14:58:47.769 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:47.768+0000 7f5d3e742640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-08-19T14:58:47.868 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:47.867+0000 7f5d3e742640 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-08-19T14:58:54.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-08-19T14:58:54.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:58:54.239 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd6 key to auth repository 2024-08-19T14:58:54.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-08-19T14:58:54.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:58:54.583 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.6 2024-08-19T14:58:54.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-08-19T14:58:54.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:58:54.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:58:54.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:58:54.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:58:54.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:54.609+0000 7f1c784d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:58:54.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:54.611+0000 7f1c784d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:58:54.613 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:54.612+0000 7f1c784d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:58:54.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-08-19T14:58:54.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:58:54.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-08-19T14:58:54.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:58:54.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:58:54.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:54.851 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:58:54.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:58:54.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:54.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:58:55.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:55.132 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:55.132+0000 7f1c784d7640 -1 Falling back to public interface 2024-08-19T14:58:56.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:56.116 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:58:56.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:56.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:58:56.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:56.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:58:56.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:57.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:57.388 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:58:57.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:57.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:58:57.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:57.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:58:57.607 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:58:57.606+0000 7f1c784d7640 -1 osd.6 0 log_to_monitors true 2024-08-19T14:58:57.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:58.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:58.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:58.709 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:58:58.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:58:58.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:58.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:58:58.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:58:59.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:58:59.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:58:59.983 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:58:59.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:58:59.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:58:59.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:59:00.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:01.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:01.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:01.264 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:59:01.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:59:01.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:01.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:59:01.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:02.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:02.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:02.710 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:59:02.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:59:02.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:02.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:59:02.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:03.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:03.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:03.987 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:59:03.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:59:03.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:03.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:59:04.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:05.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:05.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:05.266 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:59:05.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:59:05.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:05.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:59:05.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:06.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:06.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:06.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:59:06.695 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:59:06.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:06.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:59:06.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:07.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:07.972 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:59:07.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:07.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:59:07.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:07.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:59:08.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:09.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:09.247 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:59:09.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:09.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:59:09.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:09.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:59:09.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:10.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:10.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:10.783 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:59:10.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:59:10.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:10.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:59:11.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:12.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:12.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:12.361 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:59:12.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:59:12.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:12.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:59:12.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:13.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:13.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:13.983 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:59:13.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:59:13.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:13.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:59:14.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:15.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:15.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:15.762 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:59:15.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:59:15.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:15.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:59:16.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:17.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:17.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:17.547 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:59:17.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:59:17.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:17.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:59:18.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:19.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:19.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:19.007 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:59:19.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:59:19.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:19.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:59:19.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:20.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:20.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:20.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:59:20.642 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:59:20.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:20.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:59:20.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:21.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:21.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:21.921 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T14:59:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T14:59:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-08-19T14:59:22.192 INFO:tasks.workunit.client.0.smithi102.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1979154053,v1:127.0.0.1:6851/1979154053] [v2:127.0.0.1:6852/1979154053,v1:127.0.0.1:6853/1979154053] exists,up 08875c1d-be3a-41b7-9e73-295ef385a677 2024-08-19T14:59:22.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:59:22.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:59:22.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:59:22.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T14:59:22.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-08-19T14:59:22.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:59:22.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:59:22.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-08-19T14:59:22.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:59:22.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-08-19T14:59:22.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:59:22.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:59:22.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:59:22.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:59:22.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-08-19T14:59:22.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-08-19T14:59:22.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:59:22.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:59:22.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:59:22.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:59:22.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:59:22.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:59:22.199 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:59:22.200 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:59:22.200 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:59:22.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:59:22.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:59:22.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:59:22.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:59:22.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:59:22.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:59:22.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:59:22.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:59:22.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:59:22.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:59:22.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:59:22.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:59:22.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:59:22.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-08-19T14:59:22.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:59:22.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a6d10cff-a57b-48c5-a82d-791890a9bf33 2024-08-19T14:59:22.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 a6d10cff-a57b-48c5-a82d-791890a9bf33' 2024-08-19T14:59:22.202 INFO:tasks.workunit.client.0.smithi102.stdout:add osd7 a6d10cff-a57b-48c5-a82d-791890a9bf33 2024-08-19T14:59:22.202 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:59:22.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDKXcNmmCXBDBAA3a2BW09s4535OwwNJwkneA== 2024-08-19T14:59:22.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDKXcNmmCXBDBAA3a2BW09s4535OwwNJwkneA=="}' 2024-08-19T14:59:22.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a6d10cff-a57b-48c5-a82d-791890a9bf33 -i td/osd-scrub-repair/7/new.json 2024-08-19T14:59:22.576 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:59:22.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-08-19T14:59:22.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKXcNmmCXBDBAA3a2BW09s4535OwwNJwkneA== --osd-uuid a6d10cff-a57b-48c5-a82d-791890a9bf33 2024-08-19T14:59:22.616 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:59:22.615+0000 7f39c0fc4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:59:22.660 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:59:22.659+0000 7f39c0fc4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:59:22.663 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:59:22.660+0000 7f39c0fc4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:59:22.663 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:59:22.661+0000 7f39c0fc4640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-08-19T14:59:22.663 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:59:22.661+0000 7f39c0fc4640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-08-19T14:59:22.663 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:59:22.661+0000 7f39c0fc4640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-08-19T14:59:22.735 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:59:22.734+0000 7f39c0fc4640 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-08-19T14:59:28.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-08-19T14:59:28.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T14:59:28.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-08-19T14:59:28.939 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd7 key to auth repository 2024-08-19T14:59:28.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T14:59:29.293 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.7 2024-08-19T14:59:29.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-08-19T14:59:29.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T14:59:29.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T14:59:29.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T14:59:29.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T14:59:29.320 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:59:29.319+0000 7f33e7c1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:59:29.322 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:59:29.321+0000 7f33e7c1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:59:29.323 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:59:29.322+0000 7f33e7c1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:59:29.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-08-19T14:59:29.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T14:59:29.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-08-19T14:59:29.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T14:59:29.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T14:59:29.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:29.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T14:59:29.572 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T14:59:29.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:29.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:59:29.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:29.852 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:59:29.852+0000 7f33e7c1d640 -1 Falling back to public interface 2024-08-19T14:59:30.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:30.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:30.839 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T14:59:30.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T14:59:30.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:30.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:59:31.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:32.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:32.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:32.117 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T14:59:32.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T14:59:32.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:32.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:59:32.190 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:59:32.189+0000 7f33e7c1d640 -1 osd.7 0 log_to_monitors true 2024-08-19T14:59:32.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:33.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:33.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:33.392 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T14:59:33.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T14:59:33.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:33.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:59:33.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:34.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:34.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:34.791 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T14:59:34.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T14:59:34.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:34.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:59:35.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:36.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:36.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:36.167 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T14:59:36.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T14:59:36.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:36.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:59:36.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:37.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:37.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:37.444 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T14:59:37.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T14:59:37.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:37.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:59:37.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:38.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:38.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:38.839 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T14:59:38.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T14:59:38.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:38.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:59:39.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:40.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:40.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:40.144 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:59:40.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T14:59:40.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:40.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:59:40.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:41.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:41.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:41.421 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T14:59:41.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T14:59:41.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:41.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:59:41.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:42.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:42.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:42.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T14:59:42.806 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T14:59:42.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:42.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:59:43.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:44.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:44.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:44.179 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T14:59:44.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T14:59:44.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:44.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:59:44.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:45.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:45.457 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T14:59:45.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:45.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T14:59:45.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:45.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:59:46.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:47.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:47.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:47.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T14:59:47.198 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T14:59:47.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:47.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:59:47.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:48.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:48.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:48.885 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T14:59:48.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T14:59:48.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:48.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:59:49.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:50.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:50.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:50.504 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T14:59:50.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T14:59:50.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:50.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:59:50.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:51.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:51.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:51.926 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T14:59:51.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T14:59:51.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:51.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:59:52.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:53.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:53.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:53.614 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T14:59:53.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T14:59:53.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:53.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:59:53.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T14:59:54.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T14:59:54.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T14:59:54.997 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T14:59:54.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T14:59:54.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T14:59:54.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-08-19T14:59:55.277 INFO:tasks.workunit.client.0.smithi102.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/301417123,v1:127.0.0.1:6859/301417123] [v2:127.0.0.1:6860/301417123,v1:127.0.0.1:6861/301417123] exists,up a6d10cff-a57b-48c5-a82d-791890a9bf33 2024-08-19T14:59:55.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T14:59:55.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T14:59:55.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T14:59:55.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T14:59:55.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-08-19T14:59:55.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T14:59:55.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T14:59:55.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-08-19T14:59:55.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T14:59:55.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-08-19T14:59:55.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T14:59:55.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T14:59:55.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T14:59:55.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T14:59:55.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-08-19T14:59:55.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-08-19T14:59:55.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T14:59:55.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T14:59:55.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T14:59:55.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T14:59:55.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T14:59:55.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T14:59:55.284 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T14:59:55.284 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T14:59:55.284 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T14:59:55.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:59:55.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T14:59:55.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T14:59:55.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T14:59:55.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T14:59:55.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T14:59:55.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T14:59:55.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T14:59:55.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T14:59:55.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T14:59:55.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T14:59:55.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T14:59:55.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T14:59:55.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-08-19T14:59:55.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T14:59:55.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4092a65b-ff89-4c12-aa97-8c868e865e5b 2024-08-19T14:59:55.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 4092a65b-ff89-4c12-aa97-8c868e865e5b' 2024-08-19T14:59:55.285 INFO:tasks.workunit.client.0.smithi102.stdout:add osd8 4092a65b-ff89-4c12-aa97-8c868e865e5b 2024-08-19T14:59:55.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T14:59:55.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDrXcNm2vfbERAA/SLc8CTo895bYhBQYebo1A== 2024-08-19T14:59:55.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDrXcNm2vfbERAA/SLc8CTo895bYhBQYebo1A=="}' 2024-08-19T14:59:55.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4092a65b-ff89-4c12-aa97-8c868e865e5b -i td/osd-scrub-repair/8/new.json 2024-08-19T14:59:55.778 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T14:59:55.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-08-19T14:59:55.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDrXcNm2vfbERAA/SLc8CTo895bYhBQYebo1A== --osd-uuid 4092a65b-ff89-4c12-aa97-8c868e865e5b 2024-08-19T14:59:55.819 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:59:55.817+0000 7f7a7d32c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:59:55.829 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:59:55.828+0000 7f7a7d32c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:59:55.830 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:59:55.830+0000 7f7a7d32c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T14:59:55.831 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:59:55.830+0000 7f7a7d32c640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-08-19T14:59:55.831 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:59:55.830+0000 7f7a7d32c640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-08-19T14:59:55.831 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:59:55.830+0000 7f7a7d32c640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-08-19T14:59:55.942 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T14:59:55.941+0000 7f7a7d32c640 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-08-19T15:00:01.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-08-19T15:00:01.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:00:01.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-08-19T15:00:01.991 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd8 key to auth repository 2024-08-19T15:00:01.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:00:02.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-08-19T15:00:02.347 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.8 2024-08-19T15:00:02.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:00:02.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:00:02.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:00:02.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:00:02.374 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:00:02.373+0000 7f2bcea15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:00:02.376 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:00:02.375+0000 7f2bcea15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:00:02.377 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:00:02.376+0000 7f2bcea15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:00:02.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-08-19T15:00:02.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:00:02.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-08-19T15:00:02.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:00:02.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:00:02.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:02.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:00:02.616 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:00:02.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:02.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:02.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:02.899 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:00:02.898+0000 7f2bcea15640 -1 Falling back to public interface 2024-08-19T15:00:03.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:03.882 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:00:03.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:03.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:00:03.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:03.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:04.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:05.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:05.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:05.161 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:00:05.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:00:05.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:05.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:05.244 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:00:05.244+0000 7f2bcea15640 -1 osd.8 0 log_to_monitors true 2024-08-19T15:00:05.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:06.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:06.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:06.472 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:00:06.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:00:06.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:06.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:06.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:07.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:07.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:07.746 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:00:07.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:00:07.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:07.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:08.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:09.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:09.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:09.090 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:00:09.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:00:09.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:09.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:09.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:10.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:10.364 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:00:10.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:10.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:00:10.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:10.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:10.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:11.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:11.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:11.643 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:00:11.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:00:11.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:11.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:11.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:12.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:12.920 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:00:12.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:12.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:00:12.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:12.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:13.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:14.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:14.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:14.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:00:14.288 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:00:14.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:14.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:14.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:15.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:15.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:15.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:00:15.567 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:00:15.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:15.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:15.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:16.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:16.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:16.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:00:16.902 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:00:16.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:16.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:17.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:18.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:18.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:18.379 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:00:18.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:00:18.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:18.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:19.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:20.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:20.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:20.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:00:20.009 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:00:20.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:20.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:20.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:21.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:21.432 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:00:21.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:21.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:00:21.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:21.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:21.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:22.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:22.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:22.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:00:22.708 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:00:22.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:22.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:23.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:24.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:24.177 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:00:24.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:24.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:00:24.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:24.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:24.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:25.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:25.483 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:00:25.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:25.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:00:25.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:25.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:26.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:27.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:27.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:27.261 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:00:27.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:00:27.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:27.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:28.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:29.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:29.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:29.083 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T15:00:29.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T15:00:29.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:29.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-08-19T15:00:29.355 INFO:tasks.workunit.client.0.smithi102.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/4093921086,v1:127.0.0.1:6867/4093921086] [v2:127.0.0.1:6868/4093921086,v1:127.0.0.1:6869/4093921086] exists,up 4092a65b-ff89-4c12-aa97-8c868e865e5b 2024-08-19T15:00:29.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:00:29.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:00:29.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:00:29.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-08-19T15:00:29.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-08-19T15:00:29.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:00:29.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:00:29.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-08-19T15:00:29.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:00:29.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-08-19T15:00:29.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:00:29.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:00:29.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:00:29.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:00:29.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-08-19T15:00:29.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-08-19T15:00:29.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:00:29.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:00:29.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:00:29.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:00:29.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:00:29.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:00:29.361 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:00:29.361 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:00:29.361 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:00:29.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:00:29.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:00:29.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:00:29.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:00:29.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:00:29.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:00:29.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:00:29.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:00:29.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:00:29.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:00:29.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:00:29.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:00:29.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:00:29.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-08-19T15:00:29.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:00:29.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5d2f2e97-2d94-4203-81eb-651629028bc0 2024-08-19T15:00:29.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 5d2f2e97-2d94-4203-81eb-651629028bc0' 2024-08-19T15:00:29.363 INFO:tasks.workunit.client.0.smithi102.stdout:add osd9 5d2f2e97-2d94-4203-81eb-651629028bc0 2024-08-19T15:00:29.363 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:00:29.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANXsNmH6ptFhAA7Rx0pfDxOWbheVCtvGdm6w== 2024-08-19T15:00:29.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANXsNmH6ptFhAA7Rx0pfDxOWbheVCtvGdm6w=="}' 2024-08-19T15:00:29.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5d2f2e97-2d94-4203-81eb-651629028bc0 -i td/osd-scrub-repair/9/new.json 2024-08-19T15:00:29.787 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:00:29.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-08-19T15:00:29.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANXsNmH6ptFhAA7Rx0pfDxOWbheVCtvGdm6w== --osd-uuid 5d2f2e97-2d94-4203-81eb-651629028bc0 2024-08-19T15:00:29.828 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:00:29.827+0000 7f449941e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:00:29.880 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:00:29.880+0000 7f449941e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:00:29.881 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:00:29.881+0000 7f449941e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:00:29.882 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:00:29.881+0000 7f449941e640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-08-19T15:00:29.882 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:00:29.881+0000 7f449941e640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-08-19T15:00:29.882 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:00:29.881+0000 7f449941e640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-08-19T15:00:29.955 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:00:29.955+0000 7f449941e640 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-08-19T15:00:36.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-08-19T15:00:36.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:00:36.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-08-19T15:00:36.004 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd9 key to auth repository 2024-08-19T15:00:36.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:00:36.350 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.9 2024-08-19T15:00:36.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-08-19T15:00:36.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:00:36.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:00:36.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:00:36.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:00:36.377 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:00:36.375+0000 7fccc1fb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:00:36.378 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:00:36.377+0000 7fccc1fb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:00:36.380 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:00:36.378+0000 7fccc1fb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:00:36.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-08-19T15:00:36.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:00:36.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-08-19T15:00:36.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:00:36.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:00:36.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:36.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:00:36.619 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:00:36.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:36.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:00:36.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:36.895 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:00:36.894+0000 7fccc1fb6640 -1 Falling back to public interface 2024-08-19T15:00:37.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:37.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:37.889 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:00:37.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:00:37.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:37.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:00:38.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:39.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:39.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:39.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:00:39.164 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:00:39.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:39.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:00:39.256 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:00:39.255+0000 7fccc1fb6640 -1 osd.9 0 log_to_monitors true 2024-08-19T15:00:39.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:40.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:40.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:40.507 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:00:40.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:00:40.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:40.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:00:40.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:41.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:41.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:41.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:00:41.785 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:00:41.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:41.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:00:42.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:43.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:43.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:43.062 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:00:43.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:00:43.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:43.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:00:43.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:44.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:44.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:44.399 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:00:44.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:00:44.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:44.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:00:44.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:45.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:45.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:45.674 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:00:45.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:00:45.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:45.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:00:45.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:46.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:46.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:46.952 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:00:46.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:00:46.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:46.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:00:47.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:48.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:48.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:48.237 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:00:48.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:00:48.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:48.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:00:48.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:49.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:49.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:49.517 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:00:49.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:00:49.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:49.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:00:49.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:50.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:50.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:50.904 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:00:50.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:00:50.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:50.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:00:51.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:52.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:52.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:52.208 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:00:52.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:00:52.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:52.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:00:52.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:53.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:53.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:53.767 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:00:53.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:00:53.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:53.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:00:54.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:55.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:55.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:55.242 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:00:55.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:00:55.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:55.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:00:56.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:57.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:57.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:57.181 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:00:57.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:00:57.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:57.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:00:57.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:00:58.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:00:58.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:00:58.925 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:00:58.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:00:58.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:00:58.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:00:59.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:01:00.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:01:00.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:01:00.620 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:01:00.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:01:00.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:01:00.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:01:01.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:01:02.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:01:02.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:01:02.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:01:02.199 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:01:02.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:01:02.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:01:02.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:01:03.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:01:03.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:01:03.494 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T15:01:03.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T15:01:03.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:01:03.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:01:03.764 INFO:tasks.workunit.client.0.smithi102.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/3940182104,v1:127.0.0.1:6875/3940182104] [v2:127.0.0.1:6876/3940182104,v1:127.0.0.1:6877/3940182104] exists,up 5d2f2e97-2d94-4203-81eb-651629028bc0 2024-08-19T15:01:03.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:01:03.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:01:03.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:01:03.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-08-19T15:01:03.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T15:01:04.031 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-08-19T15:01:04.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T15:01:04.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T15:01:04.520 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' created 2024-08-19T15:01:04.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:01:05.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T15:01:08.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-08-19T15:01:08.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:01:08.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:01:08.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:01:08.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:01:08.669 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:01:08.669 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:01:08.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:01:08.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:01:08.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:01:08.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:01:08.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:01:08.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:01:08.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:01:08.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:01:08.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:01:09.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:01:09.053 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:01:09.053 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:01:09.053 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:01:09.053 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-08-19T15:01:09.053 INFO:tasks.workunit.client.0.smithi102.stderr:5 2024-08-19T15:01:09.053 INFO:tasks.workunit.client.0.smithi102.stderr:6 2024-08-19T15:01:09.054 INFO:tasks.workunit.client.0.smithi102.stderr:7 2024-08-19T15:01:09.054 INFO:tasks.workunit.client.0.smithi102.stderr:8 2024-08-19T15:01:09.054 INFO:tasks.workunit.client.0.smithi102.stderr:9' 2024-08-19T15:01:09.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:01:09.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:09.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:01:09.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836546 2024-08-19T15:01:09.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836546 2024-08-19T15:01:09.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546' 2024-08-19T15:01:09.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:09.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:01:09.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673020 2024-08-19T15:01:09.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673020 2024-08-19T15:01:09.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020' 2024-08-19T15:01:09.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:09.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:01:09.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509496 2024-08-19T15:01:09.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509496 2024-08-19T15:01:09.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496' 2024-08-19T15:01:09.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:09.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:01:09.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345967 2024-08-19T15:01:09.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345967 2024-08-19T15:01:09.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345967' 2024-08-19T15:01:09.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:09.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T15:01:09.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182439 2024-08-19T15:01:09.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182439 2024-08-19T15:01:09.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345967 4-107374182439' 2024-08-19T15:01:09.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:09.606 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T15:01:09.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018913 2024-08-19T15:01:09.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018913 2024-08-19T15:01:09.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345967 4-107374182439 5-128849018913' 2024-08-19T15:01:09.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:09.709 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T15:01:09.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855389 2024-08-19T15:01:09.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855389 2024-08-19T15:01:09.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345967 4-107374182439 5-128849018913 6-150323855389' 2024-08-19T15:01:09.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:09.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T15:01:09.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691858 2024-08-19T15:01:09.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691858 2024-08-19T15:01:09.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345967 4-107374182439 5-128849018913 6-150323855389 7-171798691858' 2024-08-19T15:01:09.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:09.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T15:01:10.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528334 2024-08-19T15:01:10.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528334 2024-08-19T15:01:10.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345967 4-107374182439 5-128849018913 6-150323855389 7-171798691858 8-193273528334' 2024-08-19T15:01:10.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:10.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T15:01:10.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364809 2024-08-19T15:01:10.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364809 2024-08-19T15:01:10.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345967 4-107374182439 5-128849018913 6-150323855389 7-171798691858 8-193273528334 9-214748364809' 2024-08-19T15:01:10.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:10.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836546 2024-08-19T15:01:10.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:10.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:01:10.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836546 2024-08-19T15:01:10.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:10.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836546 2024-08-19T15:01:10.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836546' 2024-08-19T15:01:10.124 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836546 2024-08-19T15:01:10.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:01:10.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836546 -lt 21474836546 2024-08-19T15:01:10.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:10.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673020 2024-08-19T15:01:10.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:10.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:01:10.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673020 2024-08-19T15:01:10.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:10.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673020 2024-08-19T15:01:10.396 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673020 2024-08-19T15:01:10.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673020' 2024-08-19T15:01:10.397 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:01:10.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673019 -lt 42949673020 2024-08-19T15:01:10.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:01:11.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:01:11.666 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:01:11.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673020 -lt 42949673020 2024-08-19T15:01:11.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:11.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509496 2024-08-19T15:01:11.939 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:11.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:01:11.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509496 2024-08-19T15:01:11.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:11.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509496 2024-08-19T15:01:11.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509496' 2024-08-19T15:01:11.941 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509496 2024-08-19T15:01:11.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:01:12.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509497 -lt 64424509496 2024-08-19T15:01:12.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:12.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345967 2024-08-19T15:01:12.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:12.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:01:12.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345967 2024-08-19T15:01:12.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:12.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345967 2024-08-19T15:01:12.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345967' 2024-08-19T15:01:12.212 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345967 2024-08-19T15:01:12.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:01:12.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345968 -lt 85899345967 2024-08-19T15:01:12.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:12.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182439 2024-08-19T15:01:12.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:12.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T15:01:12.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182439 2024-08-19T15:01:12.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:12.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182439 2024-08-19T15:01:12.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182439' 2024-08-19T15:01:12.522 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182439 2024-08-19T15:01:12.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T15:01:12.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182439 -lt 107374182439 2024-08-19T15:01:12.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:12.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018913 2024-08-19T15:01:12.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:12.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T15:01:12.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018913 2024-08-19T15:01:12.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:12.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018913 2024-08-19T15:01:12.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018913' 2024-08-19T15:01:12.796 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.5 seq 128849018913 2024-08-19T15:01:12.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T15:01:13.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018913 -lt 128849018913 2024-08-19T15:01:13.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:13.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855389 2024-08-19T15:01:13.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:13.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T15:01:13.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855389 2024-08-19T15:01:13.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:13.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855389 2024-08-19T15:01:13.073 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.6 seq 150323855389 2024-08-19T15:01:13.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855389' 2024-08-19T15:01:13.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T15:01:13.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855390 -lt 150323855389 2024-08-19T15:01:13.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:13.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691858 2024-08-19T15:01:13.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:13.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T15:01:13.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691858 2024-08-19T15:01:13.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:13.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691858 2024-08-19T15:01:13.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691858' 2024-08-19T15:01:13.348 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.7 seq 171798691858 2024-08-19T15:01:13.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T15:01:13.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691859 -lt 171798691858 2024-08-19T15:01:13.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:13.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528334 2024-08-19T15:01:13.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:13.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T15:01:13.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528334 2024-08-19T15:01:13.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:13.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528334 2024-08-19T15:01:13.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528334' 2024-08-19T15:01:13.623 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.8 seq 193273528334 2024-08-19T15:01:13.623 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T15:01:13.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528334 -lt 193273528334 2024-08-19T15:01:13.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:13.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364809 2024-08-19T15:01:13.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:13.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T15:01:13.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364809 2024-08-19T15:01:13.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:13.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364809 2024-08-19T15:01:13.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364809' 2024-08-19T15:01:13.895 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.9 seq 214748364809 2024-08-19T15:01:13.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T15:01:14.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364809 -lt 214748364809 2024-08-19T15:01:14.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:01:14.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:01:14.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:01:14.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T15:01:14.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:01:14.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:01:14.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:01:14.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:01:14.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:01:14.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:01:14.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:01:14.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:01:14.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:01:14.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:01:14.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:01:15.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T15:01:15.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:01:15.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:01:15.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2024-08-19T15:01:15.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-08-19T15:01:15.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-08-19T15:01:15.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-08-19T15:01:15.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-08-19T15:01:15.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-08-19T15:01:15.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-08-19T15:01:15.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-08-19T15:01:16.264 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'ecpool' created 2024-08-19T15:01:16.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:01:17.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-08-19T15:01:17.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-08-19T15:01:17.751 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 2 allow_ec_overwrites to true 2024-08-19T15:01:17.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-08-19T15:01:17.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:01:17.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:01:17.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:01:17.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:01:17.763 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:01:17.763 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:01:17.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:01:17.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:01:17.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:01:17.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:01:17.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:01:17.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:01:17.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:01:17.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:01:17.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:01:18.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:01:18.124 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:01:18.124 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:01:18.124 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:01:18.124 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-08-19T15:01:18.124 INFO:tasks.workunit.client.0.smithi102.stderr:5 2024-08-19T15:01:18.124 INFO:tasks.workunit.client.0.smithi102.stderr:6 2024-08-19T15:01:18.124 INFO:tasks.workunit.client.0.smithi102.stderr:7 2024-08-19T15:01:18.124 INFO:tasks.workunit.client.0.smithi102.stderr:8 2024-08-19T15:01:18.124 INFO:tasks.workunit.client.0.smithi102.stderr:9' 2024-08-19T15:01:18.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:01:18.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:18.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:01:18.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836549 2024-08-19T15:01:18.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836549 2024-08-19T15:01:18.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549' 2024-08-19T15:01:18.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:18.282 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:01:18.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673023 2024-08-19T15:01:18.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673023 2024-08-19T15:01:18.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673023' 2024-08-19T15:01:18.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:18.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:01:18.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509499 2024-08-19T15:01:18.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509499 2024-08-19T15:01:18.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509499' 2024-08-19T15:01:18.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:18.576 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:01:18.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345970 2024-08-19T15:01:18.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345970 2024-08-19T15:01:18.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509499 3-85899345970' 2024-08-19T15:01:18.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:18.678 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T15:01:18.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182442 2024-08-19T15:01:18.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182442 2024-08-19T15:01:18.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509499 3-85899345970 4-107374182442' 2024-08-19T15:01:18.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:18.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T15:01:19.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018916 2024-08-19T15:01:19.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018916 2024-08-19T15:01:19.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509499 3-85899345970 4-107374182442 5-128849018916' 2024-08-19T15:01:19.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:19.013 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T15:01:19.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855392 2024-08-19T15:01:19.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855392 2024-08-19T15:01:19.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509499 3-85899345970 4-107374182442 5-128849018916 6-150323855392' 2024-08-19T15:01:19.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:19.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T15:01:19.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691861 2024-08-19T15:01:19.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691861 2024-08-19T15:01:19.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509499 3-85899345970 4-107374182442 5-128849018916 6-150323855392 7-171798691861' 2024-08-19T15:01:19.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:19.220 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T15:01:19.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528337 2024-08-19T15:01:19.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528337 2024-08-19T15:01:19.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509499 3-85899345970 4-107374182442 5-128849018916 6-150323855392 7-171798691861 8-193273528337' 2024-08-19T15:01:19.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:19.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T15:01:19.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364812 2024-08-19T15:01:19.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364812 2024-08-19T15:01:19.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509499 3-85899345970 4-107374182442 5-128849018916 6-150323855392 7-171798691861 8-193273528337 9-214748364812' 2024-08-19T15:01:19.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:19.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836549 2024-08-19T15:01:19.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:19.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:01:19.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836549 2024-08-19T15:01:19.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:19.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836549 2024-08-19T15:01:19.504 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836549 2024-08-19T15:01:19.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836549' 2024-08-19T15:01:19.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:01:19.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836549 -lt 21474836549 2024-08-19T15:01:19.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:19.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673023 2024-08-19T15:01:19.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:19.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:01:19.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673023 2024-08-19T15:01:19.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:19.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673023 2024-08-19T15:01:19.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673023' 2024-08-19T15:01:19.896 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673023 2024-08-19T15:01:19.896 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:01:20.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673023 -lt 42949673023 2024-08-19T15:01:20.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:20.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509499 2024-08-19T15:01:20.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:20.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:01:20.203 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509499 2024-08-19T15:01:20.203 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:20.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509499 2024-08-19T15:01:20.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509499' 2024-08-19T15:01:20.204 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509499 2024-08-19T15:01:20.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:01:20.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509499 -lt 64424509499 2024-08-19T15:01:20.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:20.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345970 2024-08-19T15:01:20.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:20.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:01:20.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345970 2024-08-19T15:01:20.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:20.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345970 2024-08-19T15:01:20.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345970' 2024-08-19T15:01:20.474 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345970 2024-08-19T15:01:20.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:01:20.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345970 -lt 85899345970 2024-08-19T15:01:20.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:20.742 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182442 2024-08-19T15:01:20.742 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:20.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T15:01:20.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182442 2024-08-19T15:01:20.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:20.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182442 2024-08-19T15:01:20.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182442' 2024-08-19T15:01:20.744 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182442 2024-08-19T15:01:20.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T15:01:21.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182442 -lt 107374182442 2024-08-19T15:01:21.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:21.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018916 2024-08-19T15:01:21.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:21.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T15:01:21.013 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018916 2024-08-19T15:01:21.014 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:21.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018916 2024-08-19T15:01:21.015 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.5 seq 128849018916 2024-08-19T15:01:21.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018916' 2024-08-19T15:01:21.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T15:01:21.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018916 -lt 128849018916 2024-08-19T15:01:21.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:21.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855392 2024-08-19T15:01:21.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:21.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T15:01:21.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855392 2024-08-19T15:01:21.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:21.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855392 2024-08-19T15:01:21.331 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.6 seq 150323855392 2024-08-19T15:01:21.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855392' 2024-08-19T15:01:21.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T15:01:21.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855392 -lt 150323855392 2024-08-19T15:01:21.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:21.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691861 2024-08-19T15:01:21.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:21.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T15:01:21.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691861 2024-08-19T15:01:21.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:21.602 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.7 seq 171798691861 2024-08-19T15:01:21.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691861 2024-08-19T15:01:21.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691861' 2024-08-19T15:01:21.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T15:01:21.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691862 -lt 171798691861 2024-08-19T15:01:21.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:21.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528337 2024-08-19T15:01:21.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:21.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T15:01:21.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528337 2024-08-19T15:01:21.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:21.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528337 2024-08-19T15:01:21.872 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.8 seq 193273528337 2024-08-19T15:01:21.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528337' 2024-08-19T15:01:21.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T15:01:22.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528337 -lt 193273528337 2024-08-19T15:01:22.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:22.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364812 2024-08-19T15:01:22.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:22.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T15:01:22.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364812 2024-08-19T15:01:22.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:22.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364812 2024-08-19T15:01:22.146 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.9 seq 214748364812 2024-08-19T15:01:22.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364812' 2024-08-19T15:01:22.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T15:01:22.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364812 -lt 214748364812 2024-08-19T15:01:22.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:01:22.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:01:22.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:01:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:01:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:01:22.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:01:22.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:01:22.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:01:22.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:01:22.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:01:22.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:01:23.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:01:23.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:01:23.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:01:23.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:01:23.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:01:23.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:01:23.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:01:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-08-19T15:01:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-08-19T15:01:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-08-19T15:01:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-08-19T15:01:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-08-19T15:01:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:01:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:01:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-08-19T15:01:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:01:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:01:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:01:23.741 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:01:23.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:01:24.363 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:01:24.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:01:24.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:01:24.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-08-19T15:01:24.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-08-19T15:01:24.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-08-19T15:01:24.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T15:01:24.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-08-19T15:01:24.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:01:25.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-08-19T15:01:25.150 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-08-19T15:01:25.150 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-08-19T15:01:25.150 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-08-19T15:01:25.150 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-08-19T15:01:25.151 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-08-19T15:01:25.151 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-19T15:01:25.484 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-08-19T15:01:25.484 INFO:tasks.workunit.client.0.smithi102.stderr:5 2024-08-19T15:01:25.484 INFO:tasks.workunit.client.0.smithi102.stderr:9 2024-08-19T15:01:25.484 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:01:25.484 INFO:tasks.workunit.client.0.smithi102.stderr:6 2024-08-19T15:01:25.485 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:01:25.485 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:01:25.485 INFO:tasks.workunit.client.0.smithi102.stderr:7' 2024-08-19T15:01:25.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-08-19T15:01:25.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-08-19T15:01:25.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-08-19T15:01:25.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-08-19T15:01:25.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-08-19T15:01:25.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-08-19T15:01:25.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-08-19T15:01:25.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-08-19T15:01:25.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-08-19T15:01:25.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T15:01:25.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:01:25.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:01:25.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-08-19T15:01:25.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:01:25.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T15:01:25.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:01:25.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:01:25.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-08-19T15:01:25.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:01:25.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-08-19T15:01:25.490 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:01:25.490 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:01:25.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:01:25.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:01:25.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:01:25.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:01:25.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T15:01:25.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:01:25.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:01:25.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-08-19T15:01:25.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:01:25.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-08-19T15:01:25.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-08-19T15:01:27.922 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-08-19T15:01:28.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-08-19T15:01:28.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:01:28.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:01:28.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-19T15:01:28.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:01:28.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T15:01:28.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:01:28.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:01:28.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:01:28.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:01:28.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T15:01:28.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T15:01:28.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:01:28.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:01:28.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:01:28.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:01:28.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:01:28.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:01:28.441 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:01:28.441 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:01:28.441 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:01:28.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:01:28.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:01:28.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:01:28.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:01:28.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:01:28.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:01:28.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:01:28.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:01:28.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:01:28.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:01:28.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:01:28.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T15:01:28.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-19T15:01:28.442 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.3 2024-08-19T15:01:28.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:01:28.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-08-19T15:01:28.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-19T15:01:28.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:01:28.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:01:28.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:01:28.469 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:01:28.469+0000 7f83a1e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:01:28.472 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:01:28.471+0000 7f83a1e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:01:28.473 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:01:28.473+0000 7f83a1e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:01:28.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-19T15:01:28.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:01:28.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T15:01:28.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:01:28.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:01:28.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:01:28.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:01:28.723 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:01:28.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:01:28.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:01:28.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:01:28.995 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:01:28.995+0000 7f83a1e14640 -1 Falling back to public interface 2024-08-19T15:01:29.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:01:29.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:01:29.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:01:29.991 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:01:29.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:01:29.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:01:30.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:01:31.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:01:31.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:01:31.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:01:31.262 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:01:31.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:01:31.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:01:31.439 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:01:31.438+0000 7f83a1e14640 -1 osd.3 68 log_to_monitors true 2024-08-19T15:01:31.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:01:32.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:01:32.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:01:32.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:01:32.569 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:01:32.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:01:32.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:01:32.870 INFO:tasks.workunit.client.0.smithi102.stderr:osd.3 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [20,68) [v2:127.0.0.1:6826/3431293562,v1:127.0.0.1:6827/3431293562] [v2:127.0.0.1:6828/3431293562,v1:127.0.0.1:6829/3431293562] exists,up a53d7b50-dd2b-44ae-862b-772fc9028cb8 2024-08-19T15:01:32.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:01:32.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:01:32.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:01:32.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:01:32.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:01:32.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:01:32.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:01:32.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:01:32.871 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:01:32.871 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:01:32.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:01:32.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:01:32.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:01:32.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:01:32.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:01:32.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:01:32.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:01:32.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:01:32.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:01:33.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:01:33.261 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:01:33.261 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:01:33.261 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:01:33.261 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-08-19T15:01:33.261 INFO:tasks.workunit.client.0.smithi102.stderr:5 2024-08-19T15:01:33.261 INFO:tasks.workunit.client.0.smithi102.stderr:6 2024-08-19T15:01:33.261 INFO:tasks.workunit.client.0.smithi102.stderr:7 2024-08-19T15:01:33.261 INFO:tasks.workunit.client.0.smithi102.stderr:8 2024-08-19T15:01:33.262 INFO:tasks.workunit.client.0.smithi102.stderr:9' 2024-08-19T15:01:33.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:01:33.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:33.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:01:33.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836553 2024-08-19T15:01:33.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836553 2024-08-19T15:01:33.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553' 2024-08-19T15:01:33.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:33.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:01:33.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673027 2024-08-19T15:01:33.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673027 2024-08-19T15:01:33.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673027' 2024-08-19T15:01:33.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:33.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:01:33.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509503 2024-08-19T15:01:33.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509503 2024-08-19T15:01:33.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503' 2024-08-19T15:01:33.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:33.573 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:01:33.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645314 2024-08-19T15:01:33.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645314 2024-08-19T15:01:33.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-309237645314' 2024-08-19T15:01:33.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:33.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T15:01:33.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182446 2024-08-19T15:01:33.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182446 2024-08-19T15:01:33.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-309237645314 4-107374182446' 2024-08-19T15:01:33.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:33.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T15:01:33.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018920 2024-08-19T15:01:33.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018920 2024-08-19T15:01:33.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-309237645314 4-107374182446 5-128849018920' 2024-08-19T15:01:33.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:33.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T15:01:33.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855396 2024-08-19T15:01:33.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855396 2024-08-19T15:01:33.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-309237645314 4-107374182446 5-128849018920 6-150323855396' 2024-08-19T15:01:33.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:33.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T15:01:34.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691865 2024-08-19T15:01:34.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691865 2024-08-19T15:01:34.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-309237645314 4-107374182446 5-128849018920 6-150323855396 7-171798691865' 2024-08-19T15:01:34.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:34.091 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T15:01:34.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528341 2024-08-19T15:01:34.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528341 2024-08-19T15:01:34.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-309237645314 4-107374182446 5-128849018920 6-150323855396 7-171798691865 8-193273528341' 2024-08-19T15:01:34.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:34.196 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T15:01:34.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364816 2024-08-19T15:01:34.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364816 2024-08-19T15:01:34.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-309237645314 4-107374182446 5-128849018920 6-150323855396 7-171798691865 8-193273528341 9-214748364816' 2024-08-19T15:01:34.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:34.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836553 2024-08-19T15:01:34.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:34.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:01:34.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836553 2024-08-19T15:01:34.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:34.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836553 2024-08-19T15:01:34.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2024-08-19T15:01:34.301 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836553 2024-08-19T15:01:34.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:01:34.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836552 -lt 21474836553 2024-08-19T15:01:34.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:01:35.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:01:35.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:01:35.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836554 -lt 21474836553 2024-08-19T15:01:35.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:35.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673027 2024-08-19T15:01:35.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:35.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:01:35.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673027 2024-08-19T15:01:35.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:35.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673027 2024-08-19T15:01:35.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673027' 2024-08-19T15:01:35.847 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949673027 2024-08-19T15:01:35.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:01:36.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673027 -lt 42949673027 2024-08-19T15:01:36.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:36.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509503 2024-08-19T15:01:36.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:36.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:01:36.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509503 2024-08-19T15:01:36.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:36.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509503 2024-08-19T15:01:36.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509503' 2024-08-19T15:01:36.120 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509503 2024-08-19T15:01:36.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:01:36.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509503 -lt 64424509503 2024-08-19T15:01:36.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:36.388 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645314 2024-08-19T15:01:36.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:36.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:01:36.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645314 2024-08-19T15:01:36.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:36.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645314 2024-08-19T15:01:36.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645314' 2024-08-19T15:01:36.391 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.3 seq 309237645314 2024-08-19T15:01:36.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:01:36.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645314 -lt 309237645314 2024-08-19T15:01:36.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:36.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182446 2024-08-19T15:01:36.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:36.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T15:01:36.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182446 2024-08-19T15:01:36.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:36.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182446 2024-08-19T15:01:36.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182446' 2024-08-19T15:01:36.662 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.4 seq 107374182446 2024-08-19T15:01:36.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T15:01:36.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182446 -lt 107374182446 2024-08-19T15:01:36.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:36.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018920 2024-08-19T15:01:36.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:36.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T15:01:36.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018920 2024-08-19T15:01:36.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:36.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018920 2024-08-19T15:01:36.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018920' 2024-08-19T15:01:36.970 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.5 seq 128849018920 2024-08-19T15:01:36.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T15:01:37.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018920 -lt 128849018920 2024-08-19T15:01:37.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:37.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855396 2024-08-19T15:01:37.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:37.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T15:01:37.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855396 2024-08-19T15:01:37.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:37.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855396 2024-08-19T15:01:37.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855396' 2024-08-19T15:01:37.287 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.6 seq 150323855396 2024-08-19T15:01:37.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T15:01:37.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855397 -lt 150323855396 2024-08-19T15:01:37.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:37.579 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691865 2024-08-19T15:01:37.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:37.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T15:01:37.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691865 2024-08-19T15:01:37.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:37.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691865 2024-08-19T15:01:37.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691865' 2024-08-19T15:01:37.582 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.7 seq 171798691865 2024-08-19T15:01:37.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T15:01:37.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691866 -lt 171798691865 2024-08-19T15:01:37.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:37.849 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528341 2024-08-19T15:01:37.849 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:37.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T15:01:37.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528341 2024-08-19T15:01:37.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:37.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528341 2024-08-19T15:01:37.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528341' 2024-08-19T15:01:37.852 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.8 seq 193273528341 2024-08-19T15:01:37.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T15:01:38.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528341 -lt 193273528341 2024-08-19T15:01:38.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:38.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364816 2024-08-19T15:01:38.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:38.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T15:01:38.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364816 2024-08-19T15:01:38.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:38.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364816 2024-08-19T15:01:38.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364816' 2024-08-19T15:01:38.124 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.9 seq 214748364816 2024-08-19T15:01:38.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T15:01:38.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364816 -lt 214748364816 2024-08-19T15:01:38.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:01:38.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:01:38.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:01:38.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:01:38.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:01:38.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:01:38.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:01:38.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:01:38.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:01:38.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:01:38.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:01:38.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:01:38.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:01:38.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:01:38.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:01:39.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:01:39.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:01:39.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:01:39.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-08-19T15:01:39.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T15:01:39.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T15:01:39.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T15:01:39.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:01:39.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-08-19T15:01:39.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-08-19T15:01:39.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-08-19T15:01:39.616 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-08-19T15:01:39.616 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:01:39.616 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:01:39.616 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:01:39.616 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:01:39.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T15:01:16.129163+0000 2024-08-19T15:01:39.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-08-19T15:01:40.130 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-08-19T15:01:40.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-08-19T15:01:16.129163+0000 2024-08-19T15:01:40.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T15:01:40.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:01:16.129163+0000 2024-08-19T15:01:40.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:01:40.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:01:40.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:01:40.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:01:40.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:01:40.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:01:40.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:01:40.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:01:40.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:01:16.129163+0000 '>' 2024-08-19T15:01:16.129163+0000 2024-08-19T15:01:40.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:01:41.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:01:41.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:01:41.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:01:41.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:01:41.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:01:41.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:01:41.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:01:41.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:01:16.129163+0000 '>' 2024-08-19T15:01:16.129163+0000 2024-08-19T15:01:41.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:01:42.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:01:42.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:01:42.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:01:42.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:01:42.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:01:42.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:01:42.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:01:42.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:01:16.129163+0000 '>' 2024-08-19T15:01:16.129163+0000 2024-08-19T15:01:42.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:01:43.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:01:43.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:01:43.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:01:43.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:01:43.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:01:43.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:01:43.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:01:44.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:01:40.483478+0000 '>' 2024-08-19T15:01:16.129163+0000 2024-08-19T15:01:44.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:01:44.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T15:01:44.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:01:44.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:01:44.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-08-19T15:01:44.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:01:44.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T15:01:44.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:01:44.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:01:44.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-08-19T15:01:44.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:01:44.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-08-19T15:01:44.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:01:44.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:01:44.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:01:44.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:01:44.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:01:44.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:01:44.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T15:01:44.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:01:44.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:01:44.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-08-19T15:01:44.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:01:44.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-08-19T15:01:44.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-08-19T15:01:46.314 INFO:tasks.workunit.client.0.smithi102.stdout:_ 2024-08-19T15:01:46.314 INFO:tasks.workunit.client.0.smithi102.stdout:hinfo_key 2024-08-19T15:01:46.314 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 2024-08-19T15:01:46.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-08-19T15:01:46.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:01:46.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:01:46.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-08-19T15:01:46.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:01:46.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-08-19T15:01:46.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:01:46.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:01:46.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:01:46.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:01:46.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-08-19T15:01:46.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-08-19T15:01:46.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:01:46.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:01:46.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:01:46.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:01:46.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:01:46.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:01:46.584 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:01:46.584 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:01:46.584 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:01:46.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:01:46.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:01:46.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:01:46.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:01:46.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:01:46.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:01:46.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:01:46.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:01:46.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:01:46.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:01:46.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:01:46.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-08-19T15:01:46.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-08-19T15:01:46.585 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.9 2024-08-19T15:01:46.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:01:46.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-08-19T15:01:46.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-08-19T15:01:46.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:01:46.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:01:46.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:01:46.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:01:46.611+0000 7f37a37da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:01:46.614 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:01:46.613+0000 7f37a37da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:01:46.616 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:01:46.615+0000 7f37a37da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:01:46.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-08-19T15:01:46.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:01:46.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-08-19T15:01:46.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:01:46.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:01:46.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:01:46.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:01:46.873 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:01:46.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:01:46.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:01:47.132 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:01:47.132+0000 7f37a37da640 -1 Falling back to public interface 2024-08-19T15:01:47.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:01:48.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:01:48.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:01:48.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:01:48.141 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:01:48.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:01:48.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:01:48.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:01:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:01:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:01:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:01:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:01:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:01:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:01:49.598 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:01:49.597+0000 7f37a37da640 -1 osd.9 73 log_to_monitors true 2024-08-19T15:01:49.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:01:50.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:01:50.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:01:50.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:01:50.709 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:01:50.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:01:50.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:01:51.198 INFO:tasks.workunit.client.0.smithi102.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 74 last_clean_interval [50,73) [v2:127.0.0.1:6874/3730038857,v1:127.0.0.1:6875/3730038857] [v2:127.0.0.1:6876/3730038857,v1:127.0.0.1:6877/3730038857] exists,up 5d2f2e97-2d94-4203-81eb-651629028bc0 2024-08-19T15:01:51.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:01:51.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:01:51.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:01:51.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:01:51.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:01:51.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:01:51.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:01:51.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:01:51.199 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:01:51.199 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:01:51.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:01:51.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:01:51.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:01:51.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:01:51.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:01:51.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:01:51.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:01:51.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:01:51.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:01:51.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:01:51.635 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:01:51.635 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:01:51.635 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:01:51.635 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-08-19T15:01:51.635 INFO:tasks.workunit.client.0.smithi102.stderr:5 2024-08-19T15:01:51.636 INFO:tasks.workunit.client.0.smithi102.stderr:6 2024-08-19T15:01:51.636 INFO:tasks.workunit.client.0.smithi102.stderr:7 2024-08-19T15:01:51.636 INFO:tasks.workunit.client.0.smithi102.stderr:8 2024-08-19T15:01:51.636 INFO:tasks.workunit.client.0.smithi102.stderr:9' 2024-08-19T15:01:51.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:01:51.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:51.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:01:51.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836558 2024-08-19T15:01:51.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836558 2024-08-19T15:01:51.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558' 2024-08-19T15:01:51.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:51.752 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:01:51.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673031 2024-08-19T15:01:51.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673031 2024-08-19T15:01:51.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673031' 2024-08-19T15:01:51.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:51.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:01:52.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509508 2024-08-19T15:01:52.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509508 2024-08-19T15:01:52.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673031 2-64424509508' 2024-08-19T15:01:52.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:52.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:01:52.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645319 2024-08-19T15:01:52.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645319 2024-08-19T15:01:52.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673031 2-64424509508 3-309237645319' 2024-08-19T15:01:52.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:52.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T15:01:52.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182450 2024-08-19T15:01:52.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182450 2024-08-19T15:01:52.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673031 2-64424509508 3-309237645319 4-107374182450' 2024-08-19T15:01:52.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:52.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T15:01:52.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018925 2024-08-19T15:01:52.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018925 2024-08-19T15:01:52.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673031 2-64424509508 3-309237645319 4-107374182450 5-128849018925' 2024-08-19T15:01:52.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:52.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T15:01:52.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855401 2024-08-19T15:01:52.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855401 2024-08-19T15:01:52.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673031 2-64424509508 3-309237645319 4-107374182450 5-128849018925 6-150323855401' 2024-08-19T15:01:52.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:52.420 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T15:01:52.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691870 2024-08-19T15:01:52.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691870 2024-08-19T15:01:52.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673031 2-64424509508 3-309237645319 4-107374182450 5-128849018925 6-150323855401 7-171798691870' 2024-08-19T15:01:52.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:52.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T15:01:52.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528345 2024-08-19T15:01:52.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528345 2024-08-19T15:01:52.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673031 2-64424509508 3-309237645319 4-107374182450 5-128849018925 6-150323855401 7-171798691870 8-193273528345' 2024-08-19T15:01:52.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:01:52.623 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T15:01:52.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-08-19T15:01:52.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-08-19T15:01:52.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673031 2-64424509508 3-309237645319 4-107374182450 5-128849018925 6-150323855401 7-171798691870 8-193273528345 9-330712481794' 2024-08-19T15:01:52.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:52.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836558 2024-08-19T15:01:52.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:52.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:01:52.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836558 2024-08-19T15:01:52.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:52.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836558 2024-08-19T15:01:52.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836558' 2024-08-19T15:01:52.727 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836558 2024-08-19T15:01:52.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:01:52.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836557 -lt 21474836558 2024-08-19T15:01:52.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:01:53.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:01:53.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:01:54.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836558 -lt 21474836558 2024-08-19T15:01:54.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:54.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673031 2024-08-19T15:01:54.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:54.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:01:54.272 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673031 2024-08-19T15:01:54.272 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:54.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673031 2024-08-19T15:01:54.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673031' 2024-08-19T15:01:54.273 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949673031 2024-08-19T15:01:54.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:01:54.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673032 -lt 42949673031 2024-08-19T15:01:54.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:54.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509508 2024-08-19T15:01:54.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:54.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:01:54.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509508 2024-08-19T15:01:54.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:54.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509508 2024-08-19T15:01:54.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509508' 2024-08-19T15:01:54.545 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509508 2024-08-19T15:01:54.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:01:54.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509508 -lt 64424509508 2024-08-19T15:01:54.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:54.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645319 2024-08-19T15:01:54.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:54.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:01:54.816 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645319 2024-08-19T15:01:54.816 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:54.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645319 2024-08-19T15:01:54.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645319' 2024-08-19T15:01:54.817 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.3 seq 309237645319 2024-08-19T15:01:54.818 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:01:55.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645319 -lt 309237645319 2024-08-19T15:01:55.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:55.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182450 2024-08-19T15:01:55.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:55.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T15:01:55.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182450 2024-08-19T15:01:55.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:55.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182450 2024-08-19T15:01:55.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182450' 2024-08-19T15:01:55.085 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.4 seq 107374182450 2024-08-19T15:01:55.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T15:01:55.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182451 -lt 107374182450 2024-08-19T15:01:55.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:55.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018925 2024-08-19T15:01:55.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:55.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T15:01:55.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018925 2024-08-19T15:01:55.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:55.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018925 2024-08-19T15:01:55.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018925' 2024-08-19T15:01:55.387 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.5 seq 128849018925 2024-08-19T15:01:55.388 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T15:01:55.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018925 -lt 128849018925 2024-08-19T15:01:55.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:55.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855401 2024-08-19T15:01:55.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:55.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T15:01:55.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855401 2024-08-19T15:01:55.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:55.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855401 2024-08-19T15:01:55.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855401' 2024-08-19T15:01:55.660 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.6 seq 150323855401 2024-08-19T15:01:55.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T15:01:55.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855401 -lt 150323855401 2024-08-19T15:01:55.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:55.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691870 2024-08-19T15:01:55.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:55.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T15:01:55.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691870 2024-08-19T15:01:55.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:55.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691870 2024-08-19T15:01:55.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691870' 2024-08-19T15:01:55.933 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.7 seq 171798691870 2024-08-19T15:01:55.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T15:01:56.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691870 -lt 171798691870 2024-08-19T15:01:56.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:56.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528345 2024-08-19T15:01:56.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:56.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T15:01:56.202 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528345 2024-08-19T15:01:56.202 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:56.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528345 2024-08-19T15:01:56.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528345' 2024-08-19T15:01:56.204 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.8 seq 193273528345 2024-08-19T15:01:56.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T15:01:56.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528346 -lt 193273528345 2024-08-19T15:01:56.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:01:56.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481794 2024-08-19T15:01:56.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:01:56.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T15:01:56.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481794 2024-08-19T15:01:56.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:01:56.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-08-19T15:01:56.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-08-19T15:01:56.476 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.9 seq 330712481794 2024-08-19T15:01:56.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T15:01:56.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-08-19T15:01:56.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:01:56.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:01:56.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:01:57.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:01:57.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:01:57.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:01:57.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:01:57.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:01:57.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:01:57.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:01:57.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:01:57.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:01:57.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:01:57.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:01:57.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:01:57.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:01:57.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:01:57.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:01:57.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T15:01:57.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T15:01:57.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-08-19T15:01:57.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-08-19T15:01:57.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-08-19T15:01:57.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-08-19T15:01:57.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T15:01:57.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:01:57.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:01:57.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-08-19T15:01:57.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:01:57.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T15:01:57.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:01:57.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:01:57.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-08-19T15:01:57.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:01:57.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-08-19T15:01:57.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:01:57.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:01:57.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:01:57.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:01:57.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:01:58.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:01:58.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T15:01:58.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:01:58.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:01:58.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-08-19T15:01:58.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:01:58.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-08-19T15:01:58.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-08-19T15:02:00.134 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-08-19T15:02:00.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-08-19T15:02:00.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:02:00.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:02:00.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-08-19T15:02:00.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:02:00.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-08-19T15:02:00.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:02:00.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:02:00.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:02:00.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:02:00.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-08-19T15:02:00.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-08-19T15:02:00.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:02:00.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:02:00.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:02:00.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:02:00.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:02:00.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:02:00.654 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:02:00.654 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:02:00.654 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:02:00.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:02:00.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:02:00.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:02:00.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:02:00.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:02:00.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:02:00.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:02:00.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:02:00.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:02:00.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:02:00.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:02:00.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-08-19T15:02:00.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-08-19T15:02:00.655 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.5 2024-08-19T15:02:00.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:02:00.655 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-08-19T15:02:00.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-08-19T15:02:00.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:02:00.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:02:00.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:02:00.682 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:02:00.681+0000 7f8f2022e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:02:00.684 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:02:00.683+0000 7f8f2022e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:02:00.685 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:02:00.684+0000 7f8f2022e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:02:00.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-08-19T15:02:00.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:02:00.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-08-19T15:02:00.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:02:00.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:02:00.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:02:00.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:02:00.930 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:02:00.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:02:00.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:02:01.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:02:01.210 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:02:01.209+0000 7f8f2022e640 -1 Falling back to public interface 2024-08-19T15:02:02.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:02:02.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:02:02.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:02:02.200 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:02:02.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:02:02.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:02:02.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:02:03.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:02:03.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:02:03.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:02:03.479 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:02:03.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:02:03.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:02:03.728 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:02:03.727+0000 7f8f2022e640 -1 osd.5 79 log_to_monitors true 2024-08-19T15:02:03.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:02:04.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:02:04.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:02:04.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:02:04.747 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:02:04.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:02:04.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:02:05.024 INFO:tasks.workunit.client.0.smithi102.stderr:osd.5 up in weight 1 up_from 84 up_thru 84 down_at 80 last_clean_interval [30,79) [v2:127.0.0.1:6842/2614297011,v1:127.0.0.1:6843/2614297011] [v2:127.0.0.1:6844/2614297011,v1:127.0.0.1:6845/2614297011] exists,up 93853ae7-d7eb-460d-96c3-c71aa1567468 2024-08-19T15:02:05.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:02:05.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:02:05.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:02:05.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:02:05.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:02:05.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:02:05.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:02:05.024 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:02:05.024 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:02:05.025 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:02:05.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:02:05.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:02:05.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:02:05.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:02:05.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:02:05.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:02:05.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:02:05.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:02:05.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:02:05.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:02:05.611 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:02:05.611 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:02:05.611 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:02:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-08-19T15:02:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:5 2024-08-19T15:02:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:6 2024-08-19T15:02:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:7 2024-08-19T15:02:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:8 2024-08-19T15:02:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:9' 2024-08-19T15:02:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:02:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:05.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:02:05.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836562 2024-08-19T15:02:05.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836562 2024-08-19T15:02:05.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562' 2024-08-19T15:02:05.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:05.717 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:02:05.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673035 2024-08-19T15:02:05.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673035 2024-08-19T15:02:05.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673035' 2024-08-19T15:02:05.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:05.861 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:02:05.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509512 2024-08-19T15:02:05.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509512 2024-08-19T15:02:05.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673035 2-64424509512' 2024-08-19T15:02:05.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:05.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:02:06.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645322 2024-08-19T15:02:06.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645322 2024-08-19T15:02:06.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673035 2-64424509512 3-309237645322' 2024-08-19T15:02:06.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:06.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T15:02:06.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182454 2024-08-19T15:02:06.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182454 2024-08-19T15:02:06.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673035 2-64424509512 3-309237645322 4-107374182454' 2024-08-19T15:02:06.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:06.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T15:02:06.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252866 2024-08-19T15:02:06.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252866 2024-08-19T15:02:06.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673035 2-64424509512 3-309237645322 4-107374182454 5-360777252866' 2024-08-19T15:02:06.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:06.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T15:02:06.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855404 2024-08-19T15:02:06.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855404 2024-08-19T15:02:06.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673035 2-64424509512 3-309237645322 4-107374182454 5-360777252866 6-150323855404' 2024-08-19T15:02:06.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:06.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T15:02:06.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691874 2024-08-19T15:02:06.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691874 2024-08-19T15:02:06.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673035 2-64424509512 3-309237645322 4-107374182454 5-360777252866 6-150323855404 7-171798691874' 2024-08-19T15:02:06.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:06.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T15:02:06.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528349 2024-08-19T15:02:06.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528349 2024-08-19T15:02:06.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673035 2-64424509512 3-309237645322 4-107374182454 5-360777252866 6-150323855404 7-171798691874 8-193273528349' 2024-08-19T15:02:06.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:06.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T15:02:06.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481798 2024-08-19T15:02:06.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481798 2024-08-19T15:02:06.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673035 2-64424509512 3-309237645322 4-107374182454 5-360777252866 6-150323855404 7-171798691874 8-193273528349 9-330712481798' 2024-08-19T15:02:06.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:06.753 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836562 2024-08-19T15:02:06.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:06.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:02:06.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836562 2024-08-19T15:02:06.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:06.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836562 2024-08-19T15:02:06.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836562' 2024-08-19T15:02:06.756 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836562 2024-08-19T15:02:06.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:02:07.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836561 -lt 21474836562 2024-08-19T15:02:07.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:02:08.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:02:08.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:02:08.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836562 -lt 21474836562 2024-08-19T15:02:08.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:08.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673035 2024-08-19T15:02:08.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:08.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:02:08.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673035 2024-08-19T15:02:08.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:08.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673035 2024-08-19T15:02:08.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673035' 2024-08-19T15:02:08.306 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949673035 2024-08-19T15:02:08.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:02:08.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673036 -lt 42949673035 2024-08-19T15:02:08.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:08.573 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509512 2024-08-19T15:02:08.573 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:08.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:02:08.574 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509512 2024-08-19T15:02:08.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:08.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509512 2024-08-19T15:02:08.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509512' 2024-08-19T15:02:08.576 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509512 2024-08-19T15:02:08.576 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:02:08.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509512 -lt 64424509512 2024-08-19T15:02:08.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:08.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645322 2024-08-19T15:02:08.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:08.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:02:08.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645322 2024-08-19T15:02:08.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:08.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645322 2024-08-19T15:02:08.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645322' 2024-08-19T15:02:08.847 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.3 seq 309237645322 2024-08-19T15:02:08.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:02:09.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645323 -lt 309237645322 2024-08-19T15:02:09.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:09.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182454 2024-08-19T15:02:09.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:09.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T15:02:09.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182454 2024-08-19T15:02:09.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:09.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182454 2024-08-19T15:02:09.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182454' 2024-08-19T15:02:09.119 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.4 seq 107374182454 2024-08-19T15:02:09.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T15:02:09.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182455 -lt 107374182454 2024-08-19T15:02:09.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:09.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-360777252866 2024-08-19T15:02:09.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:09.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T15:02:09.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-360777252866 2024-08-19T15:02:09.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:09.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252866 2024-08-19T15:02:09.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 360777252866' 2024-08-19T15:02:09.411 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.5 seq 360777252866 2024-08-19T15:02:09.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T15:02:09.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252867 -lt 360777252866 2024-08-19T15:02:09.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:09.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855404 2024-08-19T15:02:09.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:09.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T15:02:09.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855404 2024-08-19T15:02:09.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:09.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855404 2024-08-19T15:02:09.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855404' 2024-08-19T15:02:09.688 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.6 seq 150323855404 2024-08-19T15:02:09.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T15:02:09.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855405 -lt 150323855404 2024-08-19T15:02:09.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:09.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691874 2024-08-19T15:02:09.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:09.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T15:02:09.959 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691874 2024-08-19T15:02:09.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:09.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691874 2024-08-19T15:02:09.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691874' 2024-08-19T15:02:09.961 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.7 seq 171798691874 2024-08-19T15:02:09.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T15:02:10.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691874 -lt 171798691874 2024-08-19T15:02:10.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:10.228 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528349 2024-08-19T15:02:10.228 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:10.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T15:02:10.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528349 2024-08-19T15:02:10.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:10.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528349 2024-08-19T15:02:10.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528349' 2024-08-19T15:02:10.231 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.8 seq 193273528349 2024-08-19T15:02:10.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T15:02:10.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528350 -lt 193273528349 2024-08-19T15:02:10.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:10.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481798 2024-08-19T15:02:10.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:10.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T15:02:10.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481798 2024-08-19T15:02:10.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:10.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481798 2024-08-19T15:02:10.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481798' 2024-08-19T15:02:10.501 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.9 seq 330712481798 2024-08-19T15:02:10.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T15:02:10.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481798 -lt 330712481798 2024-08-19T15:02:10.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:02:10.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:02:10.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:02:11.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:02:11.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:02:11.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:02:11.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:02:11.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:02:11.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:02:11.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:02:11.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:02:11.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:02:11.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:02:11.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:02:11.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:02:11.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:02:11.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:02:11.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:02:11.742 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-08-19T15:02:11.742 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T15:02:11.742 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T15:02:11.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T15:02:11.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:02:12.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-08-19T15:02:12.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-08-19T15:02:12.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-08-19T15:02:12.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-08-19T15:02:12.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:02:12.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:02:12.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:02:12.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:02:12.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T15:01:40.483478+0000 2024-08-19T15:02:12.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-08-19T15:02:12.540 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-08-19T15:02:12.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-08-19T15:01:40.483478+0000 2024-08-19T15:02:12.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T15:02:12.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:01:40.483478+0000 2024-08-19T15:02:12.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:02:12.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:02:12.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:02:12.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:02:12.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:02:12.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:02:12.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:02:12.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:02:12.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:01:40.483478+0000 '>' 2024-08-19T15:01:40.483478+0000 2024-08-19T15:02:12.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:02:13.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:02:13.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:02:13.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:02:13.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:02:13.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:02:13.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:02:13.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:02:14.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:01:40.483478+0000 '>' 2024-08-19T15:01:40.483478+0000 2024-08-19T15:02:14.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:02:15.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:02:15.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:02:15.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:02:15.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:02:15.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:02:15.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:02:15.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:02:15.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:01:40.483478+0000 '>' 2024-08-19T15:01:40.483478+0000 2024-08-19T15:02:15.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:02:16.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:02:16.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:02:16.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:02:16.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:02:16.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:02:16.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:02:16.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:02:16.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:01:40.483478+0000 '>' 2024-08-19T15:01:40.483478+0000 2024-08-19T15:02:16.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:02:17.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:02:17.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:02:17.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:02:17.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:02:17.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:02:17.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:02:17.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:02:17.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:02:13.417885+0000 '>' 2024-08-19T15:01:40.483478+0000 2024-08-19T15:02:17.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:02:17.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T15:02:17.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:02:17.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:02:17.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-08-19T15:02:17.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:02:17.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T15:02:17.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:02:17.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:02:17.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-08-19T15:02:17.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:02:17.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-08-19T15:02:17.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:02:17.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:02:17.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:02:17.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:02:17.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:02:18.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:02:18.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T15:02:18.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:02:18.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:02:18.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-08-19T15:02:18.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:02:18.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-08-19T15:02:18.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-08-19T15:02:20.028 INFO:tasks.workunit.client.0.smithi102.stdout:_ 2024-08-19T15:02:20.028 INFO:tasks.workunit.client.0.smithi102.stdout:hinfo_key 2024-08-19T15:02:20.028 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 2024-08-19T15:02:20.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-08-19T15:02:20.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:02:20.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:02:20.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-08-19T15:02:20.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:02:20.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-08-19T15:02:20.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:02:20.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:02:20.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:02:20.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:02:20.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-08-19T15:02:20.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-08-19T15:02:20.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:02:20.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:02:20.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:02:20.301 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:02:20.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:02:20.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:02:20.302 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:02:20.303 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:02:20.303 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:02:20.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:02:20.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:02:20.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:02:20.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:02:20.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:02:20.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:02:20.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:02:20.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:02:20.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:02:20.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:02:20.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:02:20.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-08-19T15:02:20.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-08-19T15:02:20.305 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.9 2024-08-19T15:02:20.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:02:20.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-08-19T15:02:20.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-08-19T15:02:20.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:02:20.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:02:20.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:02:20.327 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:02:20.327+0000 7fe64d620640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:02:20.330 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:02:20.329+0000 7fe64d620640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:02:20.332 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:02:20.331+0000 7fe64d620640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:02:20.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-08-19T15:02:20.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:02:20.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-08-19T15:02:20.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:02:20.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:02:20.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:02:20.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:02:20.583 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:02:20.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:02:20.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:02:20.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:02:20.858 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:02:20.858+0000 7fe64d620640 -1 Falling back to public interface 2024-08-19T15:02:21.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:02:21.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:02:21.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:02:21.849 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:02:21.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:02:21.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:02:22.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:02:23.114 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:02:23.113+0000 7fe64d620640 -1 osd.9 86 log_to_monitors true 2024-08-19T15:02:23.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:02:23.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:02:23.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:02:23.126 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:02:23.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:02:23.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:02:23.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:02:24.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:02:24.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:02:24.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:02:24.534 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:02:24.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:02:24.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:02:24.901 INFO:tasks.workunit.client.0.smithi102.stderr:osd.9 up in weight 1 up_from 90 up_thru 0 down_at 87 last_clean_interval [77,86) [v2:127.0.0.1:6874/2329050371,v1:127.0.0.1:6875/2329050371] [v2:127.0.0.1:6876/2329050371,v1:127.0.0.1:6877/2329050371] exists,up 5d2f2e97-2d94-4203-81eb-651629028bc0 2024-08-19T15:02:24.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:02:24.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:02:24.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:02:24.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:02:24.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:02:24.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:02:24.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:02:24.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:02:24.902 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:02:24.902 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:02:24.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:02:24.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:02:24.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:02:24.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:02:24.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:02:24.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:02:24.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:02:24.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:02:24.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:02:25.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:02:25.618 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:02:25.618 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:02:25.618 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:02:25.618 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-08-19T15:02:25.618 INFO:tasks.workunit.client.0.smithi102.stderr:5 2024-08-19T15:02:25.618 INFO:tasks.workunit.client.0.smithi102.stderr:6 2024-08-19T15:02:25.619 INFO:tasks.workunit.client.0.smithi102.stderr:7 2024-08-19T15:02:25.619 INFO:tasks.workunit.client.0.smithi102.stderr:8 2024-08-19T15:02:25.619 INFO:tasks.workunit.client.0.smithi102.stderr:9' 2024-08-19T15:02:25.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:02:25.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:25.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:02:25.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836567 2024-08-19T15:02:25.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836567 2024-08-19T15:02:25.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567' 2024-08-19T15:02:25.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:25.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:02:26.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673040 2024-08-19T15:02:26.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673040 2024-08-19T15:02:26.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673040' 2024-08-19T15:02:26.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:26.046 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:02:26.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509517 2024-08-19T15:02:26.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509517 2024-08-19T15:02:26.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509517' 2024-08-19T15:02:26.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:26.150 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:02:26.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645327 2024-08-19T15:02:26.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645327 2024-08-19T15:02:26.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509517 3-309237645327' 2024-08-19T15:02:26.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:26.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T15:02:26.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182459 2024-08-19T15:02:26.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182459 2024-08-19T15:02:26.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509517 3-309237645327 4-107374182459' 2024-08-19T15:02:26.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:26.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T15:02:26.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252871 2024-08-19T15:02:26.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252871 2024-08-19T15:02:26.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509517 3-309237645327 4-107374182459 5-360777252871' 2024-08-19T15:02:26.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:26.461 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T15:02:26.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855409 2024-08-19T15:02:26.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855409 2024-08-19T15:02:26.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509517 3-309237645327 4-107374182459 5-360777252871 6-150323855409' 2024-08-19T15:02:26.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:26.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T15:02:26.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691879 2024-08-19T15:02:26.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691879 2024-08-19T15:02:26.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509517 3-309237645327 4-107374182459 5-360777252871 6-150323855409 7-171798691879' 2024-08-19T15:02:26.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:26.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T15:02:26.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528354 2024-08-19T15:02:26.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528354 2024-08-19T15:02:26.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509517 3-309237645327 4-107374182459 5-360777252871 6-150323855409 7-171798691879 8-193273528354' 2024-08-19T15:02:26.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:26.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T15:02:26.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056642 2024-08-19T15:02:26.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056642 2024-08-19T15:02:26.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509517 3-309237645327 4-107374182459 5-360777252871 6-150323855409 7-171798691879 8-193273528354 9-386547056642' 2024-08-19T15:02:26.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:26.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836567 2024-08-19T15:02:26.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:26.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:02:26.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836567 2024-08-19T15:02:26.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:26.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836567 2024-08-19T15:02:26.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2024-08-19T15:02:26.875 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836567 2024-08-19T15:02:26.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:02:27.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836566 -lt 21474836567 2024-08-19T15:02:27.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:02:28.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:02:28.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:02:28.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836567 -lt 21474836567 2024-08-19T15:02:28.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:28.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673040 2024-08-19T15:02:28.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:28.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:02:28.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673040 2024-08-19T15:02:28.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:28.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673040 2024-08-19T15:02:28.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673040' 2024-08-19T15:02:28.420 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949673040 2024-08-19T15:02:28.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:02:28.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673041 -lt 42949673040 2024-08-19T15:02:28.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:28.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509517 2024-08-19T15:02:28.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:28.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:02:28.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509517 2024-08-19T15:02:28.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:28.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509517 2024-08-19T15:02:28.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509517' 2024-08-19T15:02:28.691 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509517 2024-08-19T15:02:28.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:02:28.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509517 -lt 64424509517 2024-08-19T15:02:28.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:28.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645327 2024-08-19T15:02:28.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:28.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:02:28.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645327 2024-08-19T15:02:28.959 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:28.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645327 2024-08-19T15:02:28.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645327' 2024-08-19T15:02:28.960 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.3 seq 309237645327 2024-08-19T15:02:28.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:02:29.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645328 -lt 309237645327 2024-08-19T15:02:29.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:29.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182459 2024-08-19T15:02:29.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:29.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T15:02:29.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182459 2024-08-19T15:02:29.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:29.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182459 2024-08-19T15:02:29.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182459' 2024-08-19T15:02:29.235 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.4 seq 107374182459 2024-08-19T15:02:29.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T15:02:29.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182460 -lt 107374182459 2024-08-19T15:02:29.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:29.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-360777252871 2024-08-19T15:02:29.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:29.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T15:02:29.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-360777252871 2024-08-19T15:02:29.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:29.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252871 2024-08-19T15:02:29.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 360777252871' 2024-08-19T15:02:29.550 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.5 seq 360777252871 2024-08-19T15:02:29.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T15:02:29.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252872 -lt 360777252871 2024-08-19T15:02:29.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:29.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855409 2024-08-19T15:02:29.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:29.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T15:02:29.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855409 2024-08-19T15:02:29.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:29.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855409 2024-08-19T15:02:29.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855409' 2024-08-19T15:02:29.824 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.6 seq 150323855409 2024-08-19T15:02:29.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T15:02:30.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855410 -lt 150323855409 2024-08-19T15:02:30.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:30.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691879 2024-08-19T15:02:30.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:30.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T15:02:30.095 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691879 2024-08-19T15:02:30.095 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:30.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691879 2024-08-19T15:02:30.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2024-08-19T15:02:30.096 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.7 seq 171798691879 2024-08-19T15:02:30.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T15:02:30.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691879 -lt 171798691879 2024-08-19T15:02:30.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:30.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528354 2024-08-19T15:02:30.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:30.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T15:02:30.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528354 2024-08-19T15:02:30.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:30.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528354 2024-08-19T15:02:30.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528354' 2024-08-19T15:02:30.367 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.8 seq 193273528354 2024-08-19T15:02:30.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T15:02:30.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528355 -lt 193273528354 2024-08-19T15:02:30.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:30.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-386547056642 2024-08-19T15:02:30.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:30.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T15:02:30.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-386547056642 2024-08-19T15:02:30.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:30.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056642 2024-08-19T15:02:30.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 386547056642' 2024-08-19T15:02:30.639 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.9 seq 386547056642 2024-08-19T15:02:30.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T15:02:30.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056643 -lt 386547056642 2024-08-19T15:02:30.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:02:30.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:02:30.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:02:31.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:02:31.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:02:31.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:02:31.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:02:31.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:02:31.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:02:31.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:02:31.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:02:31.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:02:31.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:02:31.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:02:31.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:02:31.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:02:31.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:02:31.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:02:31.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T15:02:31.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T15:02:31.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-08-19T15:02:31.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-08-19T15:02:31.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-08-19T15:02:31.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-08-19T15:02:31.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-08-19T15:02:31.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-08-19T15:02:31.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T15:02:31.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T15:02:31.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T15:02:31.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 257053"' 2024-08-19T15:02:31.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 257053' 2024-08-19T15:02:31.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-08-19T15:02:31.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T15:02:31.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T15:02:31.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 257055"' 2024-08-19T15:02:31.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T15:02:31.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 257055' 2024-08-19T15:02:31.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-08-19T15:02:31.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 257053 257055' 2024-08-19T15:02:31.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-08-19T15:02:31.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T15:02:31.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 257053 2024-08-19T15:02:31.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/257056: /' 2024-08-19T15:02:31.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-08-19T15:02:31.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/257058: /' 2024-08-19T15:02:35.027 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:02:35.028 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:02:35.028 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-08-19T15:02:35.028 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:02:35.028 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T15:02:35.028 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:02:35.028 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:02:35.028 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-08-19T15:02:35.028 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:02:35.029 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-08-19T15:02:35.029 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:02:35.029 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:02:35.029 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:02:35.029 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:02:35.029 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:02:35.029 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:02:35.029 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T15:02:35.029 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:02:35.029 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:02:35.029 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-08-19T15:02:35.029 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:02:35.029 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-08-19T15:02:35.029 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-08-19T15:02:35.030 INFO:tasks.workunit.client.0.smithi102.stderr:257056: remove 1#2:eb822e21:::SOMETHING:head# 2024-08-19T15:02:35.030 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-08-19T15:02:35.030 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:02:35.030 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:02:35.030 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-08-19T15:02:35.030 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:02:35.030 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-08-19T15:02:35.030 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:02:35.030 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:02:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:02:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:02:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-08-19T15:02:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-08-19T15:02:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:02:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:02:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:02:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:02:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:02:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:02:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:257056: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:02:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:257056: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:02:35.032 INFO:tasks.workunit.client.0.smithi102.stderr:257056: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:02:35.032 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:02:35.032 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:02:35.032 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:02:35.032 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:02:35.032 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:02:35.032 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:02:35.032 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:02:35.032 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:02:35.032 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:02:35.032 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:02:35.032 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:02:35.032 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-08-19T15:02:35.033 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-08-19T15:02:35.033 INFO:tasks.workunit.client.0.smithi102.stderr:257056: start osd.5 2024-08-19T15:02:35.033 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:02:35.067 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:02:35.067 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:02:35.067 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-08-19T15:02:35.067 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:02:35.067 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-08-19T15:02:35.067 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:02:35.067 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:02:35.067 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-08-19T15:02:35.067 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:02:35.067 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-08-19T15:02:35.067 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:02:35.067 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:02:35.068 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:02:35.068 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:02:35.068 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:02:35.068 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:02:35.068 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-08-19T15:02:35.068 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:02:35.068 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:02:35.068 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-08-19T15:02:35.068 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:02:35.068 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-08-19T15:02:35.068 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-08-19T15:02:35.068 INFO:tasks.workunit.client.0.smithi102.stderr:257058: remove 2#2:eb822e21:::SOMETHING:head# 2024-08-19T15:02:35.068 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-08-19T15:02:35.068 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:02:35.069 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:02:35.069 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-08-19T15:02:35.069 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:02:35.069 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-08-19T15:02:35.069 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:02:35.069 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:02:35.070 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:02:35.070 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:02:35.070 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-08-19T15:02:35.070 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-08-19T15:02:35.070 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:02:35.070 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:02:35.070 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:02:35.070 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:02:35.070 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:02:35.070 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:02:35.071 INFO:tasks.workunit.client.0.smithi102.stderr:257058: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:02:35.071 INFO:tasks.workunit.client.0.smithi102.stderr:257058: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:02:35.071 INFO:tasks.workunit.client.0.smithi102.stderr:257058: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:02:35.071 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:02:35.071 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:02:35.071 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:02:35.071 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:02:35.071 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:02:35.071 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:02:35.071 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:02:35.071 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:02:35.071 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:02:35.071 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:02:35.071 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:02:35.071 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-08-19T15:02:35.072 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-08-19T15:02:35.072 INFO:tasks.workunit.client.0.smithi102.stderr:257058: start osd.9 2024-08-19T15:02:35.072 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:02:39.641 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-08-19T15:02:39.641 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-08-19T15:02:39.641 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:02:39.641 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:02:39.641 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:02:39.641 INFO:tasks.workunit.client.0.smithi102.stderr:257058: 2024-08-19T15:02:35.097+0000 7f3c1f9ab640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:02:39.641 INFO:tasks.workunit.client.0.smithi102.stderr:257058: 2024-08-19T15:02:35.099+0000 7f3c1f9ab640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:02:39.641 INFO:tasks.workunit.client.0.smithi102.stderr:257058: 2024-08-19T15:02:35.100+0000 7f3c1f9ab640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:02:39.641 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-08-19T15:02:39.642 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:02:39.642 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-08-19T15:02:39.642 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:02:39.642 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:02:39.642 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:02:39.642 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:02:39.642 INFO:tasks.workunit.client.0.smithi102.stderr:257058: 0 2024-08-19T15:02:39.642 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:02:39.642 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:02:39.642 INFO:tasks.workunit.client.0.smithi102.stderr:257058: 2024-08-19T15:02:35.618+0000 7f3c1f9ab640 -1 Falling back to public interface 2024-08-19T15:02:39.642 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:02:39.642 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:02:39.643 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:02:39.643 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:02:39.643 INFO:tasks.workunit.client.0.smithi102.stderr:257058: 1 2024-08-19T15:02:39.643 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:02:39.643 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:02:39.643 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:02:39.643 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:02:39.643 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:02:39.643 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:02:39.643 INFO:tasks.workunit.client.0.smithi102.stderr:257058: 2 2024-08-19T15:02:39.643 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:02:39.643 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:02:39.643 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:02:39.644 INFO:tasks.workunit.client.0.smithi102.stderr:257058: 2024-08-19T15:02:38.465+0000 7f3c1f9ab640 -1 osd.9 92 log_to_monitors true 2024-08-19T15:02:39.644 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:02:39.644 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:02:39.644 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:02:39.644 INFO:tasks.workunit.client.0.smithi102.stderr:257058: 3 2024-08-19T15:02:39.644 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:02:39.644 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:02:39.644 INFO:tasks.workunit.client.0.smithi102.stderr:257058: osd.9 up in weight 1 up_from 97 up_thru 0 down_at 93 last_c257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-08-19T15:02:39.644 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-08-19T15:02:39.644 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:02:39.644 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:02:39.645 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:02:39.645 INFO:tasks.workunit.client.0.smithi102.stderr:257056: 2024-08-19T15:02:35.060+0000 7f541425d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:02:39.645 INFO:tasks.workunit.client.0.smithi102.stderr:257056: 2024-08-19T15:02:35.062+0000 7f541425d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:02:39.645 INFO:tasks.workunit.client.0.smithi102.stderr:257056: 2024-08-19T15:02:35.064+0000 7f541425d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:02:39.645 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-08-19T15:02:39.645 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:02:39.645 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-08-19T15:02:39.645 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:02:39.645 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:02:39.645 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:02:39.645 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:02:39.645 INFO:tasks.workunit.client.0.smithi102.stderr:257056: 0 2024-08-19T15:02:39.645 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:02:39.646 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:02:39.646 INFO:tasks.workunit.client.0.smithi102.stderr:257056: 2024-08-19T15:02:35.583+0000 7f541425d640 -1 Falling back to public interface 2024-08-19T15:02:39.646 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:02:39.646 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:02:39.646 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:02:39.646 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:02:39.646 INFO:tasks.workunit.client.0.smithi102.stderr:257056: 1 2024-08-19T15:02:39.646 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:02:39.646 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:02:39.646 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:02:39.646 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:02:39.646 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:02:39.646 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:02:39.647 INFO:tasks.workunit.client.0.smithi102.stderr:257056: 2 2024-08-19T15:02:39.647 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:02:39.647 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:02:39.647 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:02:39.647 INFO:tasks.workunit.client.0.smithi102.stderr:257056: 2024-08-19T15:02:38.235+0000 7f541425d640 -1 osd.5 92 log_to_monitors true 2024-08-19T15:02:39.647 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:02:39.647 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:02:39.647 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:02:39.647 INFO:tasks.workunit.client.0.smithi102.stderr:257056: 3 2024-08-19T15:02:39.647 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:02:39.647 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:02:40.345 INFO:tasks.workunit.client.0.smithi102.stderr:257056: osd.5 up in weight 1 up_from 97 up_thru 84 down_at 93 last_lean_interval [90,92) [v2:127.0.0.1:6874/529194266,v1:127.0.0.1:6875/529194266] [v2:127.0.0.1:6876/529194266,v1:127.0.0.1:6877/529194266] exists,up 5d2f2e97-2d94-4203-81eb-651629028bc0 2024-08-19T15:02:40.345 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:02:40.345 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:02:40.345 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:02:40.345 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:02:40.345 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:02:40.345 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:02:40.345 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:02:40.346 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:02:40.346 INFO:tasks.workunit.client.0.smithi102.stderr:257058: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:02:40.346 INFO:tasks.workunit.client.0.smithi102.stderr:257058: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:02:40.346 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:02:40.346 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:02:40.346 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:02:40.346 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:02:40.346 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:02:40.346 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:02:40.346 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:02:40.346 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:02:40.346 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:02:40.346 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:02:40.347 INFO:tasks.workunit.client.0.smithi102.stderr:257058: 1 2024-08-19T15:02:40.347 INFO:tasks.workunit.client.0.smithi102.stderr:257058: 2 2024-08-19T15:02:40.347 INFO:tasks.workunit.client.0.smithi102.stderr:257058: 3 2024-08-19T15:02:40.347 INFO:tasks.workunit.client.0.smithi102.stderr:257058: 4 2024-08-19T15:02:40.347 INFO:tasks.workunit.client.0.smithi102.stderr:257058: 5 2024-08-19T15:02:40.347 INFO:tasks.workunit.client.0.smithi102.stderr:257058: 6 2024-08-19T15:02:40.347 INFO:tasks.workunit.client.0.smithi102.stderr:257058: 7 2024-08-19T15:02:40.347 INFO:tasks.workunit.client.0.smithi102.stderr:257058: 8 2024-08-19T15:02:40.347 INFO:tasks.workunit.client.0.smithi102.stderr:257058: 9' 2024-08-19T15:02:40.347 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:02:40.347 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:40.347 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:02:40.347 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836572 2024-08-19T15:02:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836572 2024-08-19T15:02:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572' 2024-08-19T15:02:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:02:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673045 2024-08-19T15:02:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673045 2024-08-19T15:02:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673045' 2024-08-19T15:02:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/staclean_interval [84,92) [v2:127.0.0.1:6842/2393676033,v1:127.0.0.1:6843/2393676033] [v2:127.0.0.1:6844/2393676033,v1:127.0.0.1:6845/2393676033] exists,up 93853ae7-d7eb-460d-96c3-c71aa1567468 2024-08-19T15:02:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:02:40.349 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:02:40.349 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:02:40.349 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:02:40.349 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:02:40.349 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:02:40.349 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:02:40.349 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:02:40.350 INFO:tasks.workunit.client.0.smithi102.stderr:257056: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:02:40.350 INFO:tasks.workunit.client.0.smithi102.stderr:257056: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:02:40.350 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:02:40.351 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:02:40.351 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:02:40.351 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:02:40.351 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:02:40.351 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:02:40.351 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:02:40.351 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:02:40.351 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:02:40.351 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:02:40.351 INFO:tasks.workunit.client.0.smithi102.stderr:257056: 1 2024-08-19T15:02:40.351 INFO:tasks.workunit.client.0.smithi102.stderr:257056: 2 2024-08-19T15:02:40.351 INFO:tasks.workunit.client.0.smithi102.stderr:257056: 3 2024-08-19T15:02:40.351 INFO:tasks.workunit.client.0.smithi102.stderr:257056: 4 2024-08-19T15:02:40.352 INFO:tasks.workunit.client.0.smithi102.stderr:257056: 5 2024-08-19T15:02:40.352 INFO:tasks.workunit.client.0.smithi102.stderr:257056: 6 2024-08-19T15:02:40.352 INFO:tasks.workunit.client.0.smithi102.stderr:257056: 7 2024-08-19T15:02:40.352 INFO:tasks.workunit.client.0.smithi102.stderr:257056: 8 2024-08-19T15:02:40.352 INFO:tasks.workunit.client.0.smithi102.stderr:257056: 9' 2024-08-19T15:02:40.352 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:02:40.352 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:40.352 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:02:40.353 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836571 2024-08-19T15:02:40.353 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836571 2024-08-19T15:02:40.353 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571' 2024-08-19T15:02:40.353 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:40.353 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:02:40.353 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673044 2024-08-19T15:02:40.353 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673044 2024-08-19T15:02:40.353 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673044' 2024-08-19T15:02:40.353 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:41.702 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qndalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:02:41.703 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509521 2024-08-19T15:02:41.703 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509521 2024-08-19T15:02:41.703 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509521' 2024-08-19T15:02:41.703 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:41.703 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:02:41.703 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645331 2024-08-19T15:02:41.703 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645331 2024-08-19T15:02:41.703 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509521 3-309237645331' 2024-08-19T15:02:41.703 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:41.703 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T15:02:41.703 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182463 2024-08-19T15:02:41.704 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182463 2024-08-19T15:02:41.704 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509521 3-309237645331 4-107374182463' 2024-08-19T15:02:41.704 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:41.704 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T15:02:41.704 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827714 2024-08-19T15:02:41.704 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827714 2024-08-19T15:02:41.704 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509521 3-309237645331 4-107374182463 5-416611827714' 2024-08-19T15:02:41.704 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:41.704 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T15:02:41.704 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855414 2024-08-19T15:02:41.704 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855414 2024-08-19T15:02:41.704 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509521 3-309237645331 4-107374182463 5-416611827714 6-150323855414' 2024-08-19T15:02:41.704 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:41.705 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T15:02:41.705 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691884 2024-08-19T15:02:41.705 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691884 2024-08-19T15:02:41.705 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509521 3-309237645331 4-107374182463 5-416611827714 6-150323855414 7-171798691884' 2024-08-19T15:02:41.705 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:41.705 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T15:02:41.705 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.clia/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:02:41.705 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509522 2024-08-19T15:02:41.705 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509522 2024-08-19T15:02:41.705 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673044 2-64424509522' 2024-08-19T15:02:41.705 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:41.705 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:02:41.706 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645332 2024-08-19T15:02:41.706 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645332 2024-08-19T15:02:41.706 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673044 2-64424509522 3-309237645332' 2024-08-19T15:02:41.706 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:41.706 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T15:02:41.706 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182464 2024-08-19T15:02:41.706 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182464 2024-08-19T15:02:41.706 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673044 2-64424509522 3-309237645332 4-107374182464' 2024-08-19T15:02:41.706 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:41.706 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T15:02:41.706 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827715 2024-08-19T15:02:41.706 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827715 2024-08-19T15:02:41.706 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673044 2-64424509522 3-309237645332 4-107374182464 5-416611827715' 2024-08-19T15:02:41.707 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:41.707 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T15:02:41.707 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855413 2024-08-19T15:02:41.707 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855413 2024-08-19T15:02:41.707 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673044 2-64424509522 3-309237645332 4-107374182464 5-416611827715 6-150323855413' 2024-08-19T15:02:41.707 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:41.707 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T15:02:41.707 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691883 2024-08-19T15:02:41.707 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691883 2024-08-19T15:02:41.707 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673044 2-64424509522 3-309237645332 4-107374182464 5-416611827715 6-150323855413 7-171798691883' 2024-08-19T15:02:41.707 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:41.707 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T15:02:42.375 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528358 2024-08-19T15:02:42.375 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528358 2024-08-19T15:02:42.375 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673044 2-64424509522 3-309237645332 4-107374182464 5-416611827715 6-150323855413 7-171798691883 8-193273528358' 2024-08-19T15:02:42.375 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:42.375 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T15:02:42.375 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827715 2024-08-19T15:02:42.375 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827715 2024-08-19T15:02:42.375 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673044 2-64424509522 3-309237645332 4-107374182464 5-416611827715 6-150323855413 7-171798691883 8-193273528358 9-416611827715' 2024-08-19T15:02:42.375 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:42.375 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836571 2024-08-19T15:02:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:02:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836571 2024-08-19T15:02:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836571 2024-08-19T15:02:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836571' 2024-08-19T15:02:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:257056: waiting osd.0 seq 21474836571 2024-08-19T15:02:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:02:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836572 -lt 21474836571 2024-08-19T15:02:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673044 2024-08-19T15:02:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:02:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673044 2024-08-19T15:02:42.377 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:42.377 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673044 2024-08-19T15:02:42.377 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673044' 2024-08-19T15:02:42.377 INFO:tasks.workunit.client.0.smithi102.stderr:257056: waiting osd.1 seq 42949673044 2024-08-19T15:02:42.377 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:02:42.377 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673045 -lt 42949673044 2024-08-19T15:02:42.377 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:42.377 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509522 2024-08-19T15:02:42.377 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:42.377 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:02:42.384 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.ent.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528359 2024-08-19T15:02:42.384 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528359 2024-08-19T15:02:42.384 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509521 3-309237645331 4-107374182463 5-416611827714 6-150323855414 7-171798691884 8-193273528359' 2024-08-19T15:02:42.384 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:02:42.384 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T15:02:42.384 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827714 2024-08-19T15:02:42.384 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827714 2024-08-19T15:02:42.384 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509521 3-309237645331 4-107374182463 5-416611827714 6-150323855414 7-171798691884 8-193273528359 9-416611827714' 2024-08-19T15:02:42.385 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:42.385 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836572 2024-08-19T15:02:42.385 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:42.385 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:02:42.385 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836572 2024-08-19T15:02:42.385 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:42.386 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836572 2024-08-19T15:02:42.386 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836572' 2024-08-19T15:02:42.386 INFO:tasks.workunit.client.0.smithi102.stderr:257058: waiting osd.0 seq 21474836572 2024-08-19T15:02:42.386 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:02:42.386 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836572 -lt 21474836572 2024-08-19T15:02:42.386 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:42.386 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673045 2024-08-19T15:02:42.386 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:42.386 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:02:42.386 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673045 2024-08-19T15:02:42.386 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:42.386 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673045 2024-08-19T15:02:42.386 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673045' 2024-08-19T15:02:42.387 INFO:tasks.workunit.client.0.smithi102.stderr:257058: waiting osd.1 seq 42949673045 2024-08-19T15:02:42.387 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:02:42.387 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673045 -lt 42949673045 2024-08-19T15:02:42.387 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:42.387 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509521 2024-08-19T15:02:42.387 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:42.387 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:02:43.285 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.clienclient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509522 2024-08-19T15:02:43.285 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:43.286 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509522 2024-08-19T15:02:43.286 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509522' 2024-08-19T15:02:43.286 INFO:tasks.workunit.client.0.smithi102.stderr:257056: waiting osd.2 seq 64424509522 2024-08-19T15:02:43.286 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:02:43.286 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509522 -lt 64424509522 2024-08-19T15:02:43.286 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:43.286 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645332 2024-08-19T15:02:43.286 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:43.286 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:02:43.286 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645332 2024-08-19T15:02:43.286 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:43.287 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645332 2024-08-19T15:02:43.287 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645332' 2024-08-19T15:02:43.287 INFO:tasks.workunit.client.0.smithi102.stderr:257056: waiting osd.3 seq 309237645332 2024-08-19T15:02:43.287 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:02:43.287 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645332 -lt 309237645332 2024-08-19T15:02:43.287 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:43.287 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182464 2024-08-19T15:02:43.287 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:43.287 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T15:02:43.287 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182464 2024-08-19T15:02:43.287 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:43.287 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182464 2024-08-19T15:02:43.287 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182464' 2024-08-19T15:02:43.288 INFO:tasks.workunit.client.0.smithi102.stderr:257056: waiting osd.4 seq 107374182464 2024-08-19T15:02:43.288 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T15:02:43.288 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182464 -lt 107374182464 2024-08-19T15:02:43.288 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:43.288 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-416611827715 2024-08-19T15:02:43.288 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:43.288 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T15:02:43.288 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-416611827715 2024-08-19T15:02:43.288 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:43.288 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827715 2024-08-19T15:02:43.289 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waitt.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509521 2024-08-19T15:02:43.289 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:43.289 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509521 2024-08-19T15:02:43.289 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509521' 2024-08-19T15:02:43.289 INFO:tasks.workunit.client.0.smithi102.stderr:257058: waiting osd.2 seq 64424509521 2024-08-19T15:02:43.289 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:02:43.289 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509522 -lt 64424509521 2024-08-19T15:02:43.289 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:43.289 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645331 2024-08-19T15:02:43.289 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:43.289 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:02:43.290 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645331 2024-08-19T15:02:43.290 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:43.290 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645331 2024-08-19T15:02:43.290 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645331' 2024-08-19T15:02:43.290 INFO:tasks.workunit.client.0.smithi102.stderr:257058: waiting osd.3 seq 309237645331 2024-08-19T15:02:43.290 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:02:43.290 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645332 -lt 309237645331 2024-08-19T15:02:43.290 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:43.290 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182463 2024-08-19T15:02:43.290 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:43.290 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T15:02:43.290 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182463 2024-08-19T15:02:43.290 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:43.291 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182463 2024-08-19T15:02:43.291 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182463' 2024-08-19T15:02:43.291 INFO:tasks.workunit.client.0.smithi102.stderr:257058: waiting osd.4 seq 107374182463 2024-08-19T15:02:43.291 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T15:02:43.291 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182464 -lt 107374182463 2024-08-19T15:02:43.291 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:43.291 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-416611827714 2024-08-19T15:02:43.291 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:43.291 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T15:02:43.291 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-416611827714 2024-08-19T15:02:43.291 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:43.291 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827714 2024-08-19T15:02:44.431 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 416611827714' 2024-08-19T15:02:44.431 INFO:tasks.workunit.client.0.smithi102.stderr:257058: waiting osd.5 seq 416611827714 2024-08-19T15:02:44.431 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T15:02:44.432 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827715 -lt 416611827714 2024-08-19T15:02:44.432 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:44.432 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855414 2024-08-19T15:02:44.432 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:44.432 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T15:02:44.432 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855414 2024-08-19T15:02:44.432 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:44.432 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855414 2024-08-19T15:02:44.432 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855414' 2024-08-19T15:02:44.432 INFO:tasks.workunit.client.0.smithi102.stderr:257058: waiting osd.6 seq 150323855414 2024-08-19T15:02:44.432 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T15:02:44.432 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855415 -lt 150323855414 2024-08-19T15:02:44.432 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:44.433 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691884 2024-08-19T15:02:44.433 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:44.433 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T15:02:44.433 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691884 2024-08-19T15:02:44.433 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:44.433 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691884 2024-08-19T15:02:44.433 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691884' 2024-08-19T15:02:44.433 INFO:tasks.workunit.client.0.smithi102.stderr:257058: waiting osd.7 seq 171798691884 2024-08-19T15:02:44.433 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T15:02:44.433 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691884 -lt 171798691884 2024-08-19T15:02:44.433 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:44.433 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528359 2024-08-19T15:02:44.434 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:44.434 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T15:02:44.434 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528359 2024-08-19T15:02:44.434 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:44.434 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528359 2024-08-19T15:02:44.434 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528359' 2024-08-19T15:02:44.434 INFO:tasks.workunit.client.0.smithi102.stderr:257058: waiting osd.8 seq 193273528359 2024-08-19T15:02:44.434 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T15:02:44.434 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528359 -lt 193273528359 2024-08-19T15:02:44.523 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_staing osd.5 seq 416611827715' 2024-08-19T15:02:44.523 INFO:tasks.workunit.client.0.smithi102.stderr:257056: waiting osd.5 seq 416611827715 2024-08-19T15:02:44.523 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T15:02:44.523 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827715 -lt 416611827715 2024-08-19T15:02:44.523 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:44.523 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855413 2024-08-19T15:02:44.523 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:44.523 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T15:02:44.523 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855413 2024-08-19T15:02:44.523 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:44.523 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855413 2024-08-19T15:02:44.523 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855413' 2024-08-19T15:02:44.524 INFO:tasks.workunit.client.0.smithi102.stderr:257056: waiting osd.6 seq 150323855413 2024-08-19T15:02:44.524 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T15:02:44.524 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855415 -lt 150323855413 2024-08-19T15:02:44.524 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:44.524 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691883 2024-08-19T15:02:44.524 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:44.524 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T15:02:44.524 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691883 2024-08-19T15:02:44.524 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:44.524 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691883 2024-08-19T15:02:44.524 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691883' 2024-08-19T15:02:44.524 INFO:tasks.workunit.client.0.smithi102.stderr:257056: waiting osd.7 seq 171798691883 2024-08-19T15:02:44.524 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T15:02:44.525 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691884 -lt 171798691883 2024-08-19T15:02:44.525 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:44.525 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528358 2024-08-19T15:02:44.525 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:44.525 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T15:02:44.525 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528358 2024-08-19T15:02:44.525 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:44.525 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528358 2024-08-19T15:02:44.525 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528358' 2024-08-19T15:02:44.525 INFO:tasks.workunit.client.0.smithi102.stderr:257056: waiting osd.8 seq 193273528358 2024-08-19T15:02:44.525 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T15:02:44.525 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528359 -lt 193273528358 2024-08-19T15:02:45.835 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:02:45.835 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-416611827715 2024-08-19T15:02:45.835 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:45.835 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T15:02:45.835 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-416611827715 2024-08-19T15:02:45.835 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:45.835 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827715 2024-08-19T15:02:45.836 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 416611827715' 2024-08-19T15:02:45.836 INFO:tasks.workunit.client.0.smithi102.stderr:257056: waiting osd.9 seq 416611827715 2024-08-19T15:02:45.836 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T15:02:45.836 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827715 -lt 416611827715 2024-08-19T15:02:45.836 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:02:45.836 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:02:45.836 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:02:45.836 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:02:45.836 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:02:45.836 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:02:45.836 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:02:45.836 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:02:45.836 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:02:45.837 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:02:45.837 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:02:45.837 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:02:45.837 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:02:45.837 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:02:45.837 INFO:tasks.workunit.client.0.smithi102.stderr:257056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:02:45.837 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:02:45.837 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:02:45.837 INFO:tasks.workunit.client.0.smithi102.stderr:257056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:02:45.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T15:02:45.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T15:02:45.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 257055 2024-08-19T15:02:45.839 INFO:tasks.workunit.client.0.smithi102.stderr:ts: for s in $seqs 2024-08-19T15:02:45.839 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-416611827714 2024-08-19T15:02:45.839 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:02:45.839 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T15:02:45.839 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-416611827714 2024-08-19T15:02:45.839 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:02:45.839 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827714 2024-08-19T15:02:45.839 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 416611827714' 2024-08-19T15:02:45.839 INFO:tasks.workunit.client.0.smithi102.stderr:257058: waiting osd.9 seq 416611827714 2024-08-19T15:02:45.839 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T15:02:45.839 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827715 -lt 416611827714 2024-08-19T15:02:45.839 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:02:45.840 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:02:45.840 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:02:45.840 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:02:45.840 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:02:45.840 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:02:45.840 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:02:45.840 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:02:45.840 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:02:45.840 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:02:45.840 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:02:45.840 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:02:45.840 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:02:45.840 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:02:45.841 INFO:tasks.workunit.client.0.smithi102.stderr:257058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:02:45.841 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:02:45.841 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:02:45.841 INFO:tasks.workunit.client.0.smithi102.stderr:257058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:02:45.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T15:02:45.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-08-19T15:02:45.841 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-08-19T15:02:45.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-08-19T15:02:45.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-08-19T15:02:45.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-08-19T15:02:45.841 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-08-19T15:02:45.841 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T15:02:45.841 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T15:02:45.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T15:02:45.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:02:46.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-08-19T15:02:46.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-08-19T15:02:46.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-08-19T15:02:46.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-08-19T15:02:46.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:02:46.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:02:46.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:02:46.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:02:46.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T15:02:13.417885+0000 2024-08-19T15:02:46.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-08-19T15:02:46.683 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-08-19T15:02:46.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-08-19T15:02:13.417885+0000 2024-08-19T15:02:46.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T15:02:46.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:02:13.417885+0000 2024-08-19T15:02:46.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:02:46.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:02:46.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:02:46.695 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:02:46.695 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:02:46.695 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:02:46.695 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:02:46.695 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:02:46.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:02:13.417885+0000 '>' 2024-08-19T15:02:13.417885+0000 2024-08-19T15:02:46.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:02:47.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:02:47.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:02:47.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:02:47.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:02:47.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:02:47.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:02:47.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:02:48.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:02:13.417885+0000 '>' 2024-08-19T15:02:13.417885+0000 2024-08-19T15:02:48.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:02:49.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:02:49.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:02:49.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:02:49.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:02:49.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:02:49.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:02:49.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:02:49.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:02:13.417885+0000 '>' 2024-08-19T15:02:13.417885+0000 2024-08-19T15:02:49.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:02:50.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:02:50.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:02:50.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:02:50.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:02:50.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:02:50.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:02:50.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:02:50.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:02:13.417885+0000 '>' 2024-08-19T15:02:13.417885+0000 2024-08-19T15:02:50.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:02:51.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:02:51.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:02:51.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:02:51.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:02:51.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:02:51.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:02:51.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:02:52.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:02:13.417885+0000 '>' 2024-08-19T15:02:13.417885+0000 2024-08-19T15:02:52.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:02:53.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:02:53.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:02:53.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:02:53.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:02:53.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:02:53.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:02:53.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:02:53.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:02:47.305328+0000 '>' 2024-08-19T15:02:13.417885+0000 2024-08-19T15:02:53.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:02:53.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-08-19T15:02:53.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-08-19T15:02:53.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T15:02:53.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T15:02:53.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 260484"' 2024-08-19T15:02:53.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 260484' 2024-08-19T15:02:53.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T15:02:53.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T15:02:53.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T15:02:53.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 260486"' 2024-08-19T15:02:53.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 260486' 2024-08-19T15:02:53.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-08-19T15:02:53.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-08-19T15:02:53.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 260484 260486' 2024-08-19T15:02:53.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-08-19T15:02:53.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T15:02:53.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 260484 2024-08-19T15:02:53.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/260487: /' 2024-08-19T15:02:53.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T15:02:53.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/260489: /' 2024-08-19T15:02:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:02:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:02:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-08-19T15:02:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:02:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T15:02:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:02:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:02:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-08-19T15:02:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:02:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-08-19T15:02:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:02:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:02:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:02:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:02:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:02:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:02:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-08-19T15:02:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:02:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:02:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-08-19T15:02:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:02:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-08-19T15:02:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-08-19T15:02:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:260489: _ 2024-08-19T15:02:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:260489: hinfo_key 2024-08-19T15:02:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:260489: snapset 2024-08-19T15:02:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-08-19T15:02:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:02:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:02:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-08-19T15:02:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:02:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-08-19T15:02:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:02:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:02:55.983 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:02:55.983 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:02:55.983 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-08-19T15:02:55.983 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-08-19T15:02:55.983 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:02:55.983 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:02:55.983 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:02:55.983 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:02:55.984 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:02:55.984 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:02:55.984 INFO:tasks.workunit.client.0.smithi102.stderr:260489: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:02:55.984 INFO:tasks.workunit.client.0.smithi102.stderr:260489: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:02:55.984 INFO:tasks.workunit.client.0.smithi102.stderr:260489: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:02:55.984 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:02:55.984 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:02:55.984 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:02:55.984 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:02:55.984 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:02:55.984 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:02:55.984 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:02:55.984 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:02:55.985 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:02:55.985 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:02:55.985 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:02:55.985 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-08-19T15:02:55.985 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-08-19T15:02:55.985 INFO:tasks.workunit.client.0.smithi102.stderr:260489: start osd.9 2024-08-19T15:02:56.314 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:02:56.315 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:02:56.315 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-08-19T15:02:56.315 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:02:56.315 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-08-19T15:02:56.315 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:02:56.315 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:02:56.315 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-08-19T15:02:56.315 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:02:56.315 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-08-19T15:02:56.315 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:02:56.315 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:02:56.316 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:02:56.316 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:02:56.316 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:02:56.316 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:02:56.316 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-08-19T15:02:56.316 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:02:56.316 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:02:56.316 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-08-19T15:02:56.316 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:02:56.316 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-08-19T15:02:56.316 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-08-19T15:02:56.316 INFO:tasks.workunit.client.0.smithi102.stderr:260487: _ 2024-08-19T15:02:56.316 INFO:tasks.workunit.client.0.smithi102.stderr:260487: hinfo_key 2024-08-19T15:02:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:260487: snapset 2024-08-19T15:02:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-08-19T15:02:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:02:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:02:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-08-19T15:02:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:02:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-08-19T15:02:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:02:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:02:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:02:56.318 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:02:56.318 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-08-19T15:02:56.318 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-08-19T15:02:56.318 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:02:56.318 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:02:56.318 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:02:56.318 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:02:56.318 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:02:56.318 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:02:56.318 INFO:tasks.workunit.client.0.smithi102.stderr:260487: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:02:56.318 INFO:tasks.workunit.client.0.smithi102.stderr:260487: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:02:56.318 INFO:tasks.workunit.client.0.smithi102.stderr:260487: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:02:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:02:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:02:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:02:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:02:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:02:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:02:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:02:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:02:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:02:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:02:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:02:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-08-19T15:02:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-08-19T15:02:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:260487: start osd.5 2024-08-19T15:03:00.384 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:03:00.384 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-08-19T15:03:00.384 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-08-19T15:03:00.384 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:03:00.384 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:03:00.385 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:03:00.385 INFO:tasks.workunit.client.0.smithi102.stderr:260489: 2024-08-19T15:02:56.013+0000 7f6e831a7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:00.385 INFO:tasks.workunit.client.0.smithi102.stderr:260489: 2024-08-19T15:02:56.016+0000 7f6e831a7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:00.385 INFO:tasks.workunit.client.0.smithi102.stderr:260489: 2024-08-19T15:02:56.017+0000 7f6e831a7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:00.385 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-08-19T15:03:00.385 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:03:00.385 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-08-19T15:03:00.385 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:03:00.385 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:03:00.385 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:00.385 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:03:00.385 INFO:tasks.workunit.client.0.smithi102.stderr:260489: 0 2024-08-19T15:03:00.385 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:00.385 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:03:00.386 INFO:tasks.workunit.client.0.smithi102.stderr:260489: 2024-08-19T15:02:56.532+0000 7f6e831a7640 -1 Falling back to public interface 2024-08-19T15:03:00.386 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:00.386 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:00.386 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:00.386 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:03:00.386 INFO:tasks.workunit.client.0.smithi102.stderr:260489: 1 2024-08-19T15:03:00.386 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:00.386 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:03:00.386 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:00.386 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:00.386 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:00.386 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:03:00.387 INFO:tasks.workunit.client.0.smithi102.stderr:260489: 2 2024-08-19T15:03:00.387 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:00.387 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:03:00.387 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:00.387 INFO:tasks.workunit.client.0.smithi102.stderr:260489: 2024-08-19T15:02:59.326+0000 7f6e831a7640 -1 osd.9 99 log_to_monitors true 2024-08-19T15:03:00.387 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:00.387 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:00.387 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:03:00.387 INFO:tasks.workunit.client.0.smithi102.stderr:260489: 3 2024-08-19T15:03:00.387 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:00.387 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-08-19T15:03:00.752 INFO:tasks.workunit.client.0.smithi102.stderr:260489: osd.9 up in weight 1 up_from 103 up_thru 0 down_aovery_ops 2024-08-19T15:03:00.752 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-08-19T15:03:00.752 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-08-19T15:03:00.752 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:03:00.752 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:03:00.752 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:03:00.752 INFO:tasks.workunit.client.0.smithi102.stderr:260487: 2024-08-19T15:02:56.343+0000 7fb29ad1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:00.752 INFO:tasks.workunit.client.0.smithi102.stderr:260487: 2024-08-19T15:02:56.347+0000 7fb29ad1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:00.752 INFO:tasks.workunit.client.0.smithi102.stderr:260487: 2024-08-19T15:02:56.348+0000 7fb29ad1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:00.752 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-08-19T15:03:00.753 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:03:00.753 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-08-19T15:03:00.753 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:03:00.753 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:03:00.753 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:00.753 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:03:00.753 INFO:tasks.workunit.client.0.smithi102.stderr:260487: 0 2024-08-19T15:03:00.753 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:00.753 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:03:00.753 INFO:tasks.workunit.client.0.smithi102.stderr:260487: 2024-08-19T15:02:56.864+0000 7fb29ad1a640 -1 Falling back to public interface 2024-08-19T15:03:00.753 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:00.753 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:00.753 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:00.754 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:03:00.754 INFO:tasks.workunit.client.0.smithi102.stderr:260487: 1 2024-08-19T15:03:00.754 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:00.754 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:03:00.754 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:00.754 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:00.754 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:00.754 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:03:00.754 INFO:tasks.workunit.client.0.smithi102.stderr:260487: 2 2024-08-19T15:03:00.754 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:00.754 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:03:00.754 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:00.754 INFO:tasks.workunit.client.0.smithi102.stderr:260487: 2024-08-19T15:02:59.887+0000 7fb29ad1a640 -1 osd.5 99 log_to_monitors true 2024-08-19T15:03:00.755 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:00.755 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:00.755 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:03:00.755 INFO:tasks.workunit.client.0.smithi102.stderr:260487: 3 2024-08-19T15:03:00.755 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:00.755 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:03:01.258 INFO:tasks.workunit.client.0.smithi102.stderr:260487: osd.5 up in weight 1 up_from 104 up_thru 97 down_t 100 last_clean_interval [97,99) [v2:127.0.0.1:6842/4120580283,v1:127.0.0.1:6843/4120580283] [v2:127.0.0.1:6844/4120580283,v1:127.0.0.1:6845/4120580283] exists,up 5d2f2e97-2d94-4203-81eb-651629028bc0 2024-08-19T15:03:01.258 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:03:01.258 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:03:01.258 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:03:01.258 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:03:01.258 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:03:01.258 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:03:01.259 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:03:01.259 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:03:01.259 INFO:tasks.workunit.client.0.smithi102.stderr:260489: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:03:01.259 INFO:tasks.workunit.client.0.smithi102.stderr:260489: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:03:01.259 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:03:01.259 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:03:01.259 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:03:01.259 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:03:01.259 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:03:01.259 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:03:01.259 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:03:01.259 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:03:01.259 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:03:01.260 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:03:01.260 INFO:tasks.workunit.client.0.smithi102.stderr:260489: 1 2024-08-19T15:03:01.260 INFO:tasks.workunit.client.0.smithi102.stderr:260489: 2 2024-08-19T15:03:01.260 INFO:tasks.workunit.client.0.smithi102.stderr:260489: 3 2024-08-19T15:03:01.260 INFO:tasks.workunit.client.0.smithi102.stderr:260489: 4 2024-08-19T15:03:01.260 INFO:tasks.workunit.client.0.smithi102.stderr:260489: 5 2024-08-19T15:03:01.260 INFO:tasks.workunit.client.0.smithi102.stderr:260489: 6 2024-08-19T15:03:01.260 INFO:tasks.workunit.client.0.smithi102.stderr:260489: 7 2024-08-19T15:03:01.260 INFO:tasks.workunit.client.0.smithi102.stderr:260489: 8 2024-08-19T15:03:01.260 INFO:tasks.workunit.client.0.smithi102.stderr:260489: 9' 2024-08-19T15:03:01.260 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:03:01.260 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:01.260 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:03:01.260 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836577 2024-08-19T15:03:01.261 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836577 2024-08-19T15:03:01.261 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577' 2024-08-19T15:03:01.261 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:01.261 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:03:01.261 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673050 2024-08-19T15:03:01.261 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673050 2024-08-19T15:03:01.261 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673050' 2024-08-19T15:03:01.261 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:01.606 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/cloneat 100 last_clean_interval [97,99) [v2:127.0.0.1:6874/1764397380,v1:127.0.0.1:6875/1764397380] [v2:127.0.0.1:6876/1764397380,v1:127.0.0.1:6877/1764397380] exists,up 93853ae7-d7eb-460d-96c3-c71aa1567468 2024-08-19T15:03:01.607 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:03:01.607 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:03:01.607 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:03:01.607 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:03:01.607 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:03:01.607 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:03:01.607 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:03:01.607 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:03:01.607 INFO:tasks.workunit.client.0.smithi102.stderr:260487: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:03:01.607 INFO:tasks.workunit.client.0.smithi102.stderr:260487: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:03:01.607 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:03:01.608 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:03:01.608 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:03:01.608 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:03:01.608 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:03:01.608 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:03:01.608 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:03:01.608 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:03:01.608 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:03:01.608 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:03:01.608 INFO:tasks.workunit.client.0.smithi102.stderr:260487: 1 2024-08-19T15:03:01.608 INFO:tasks.workunit.client.0.smithi102.stderr:260487: 2 2024-08-19T15:03:01.608 INFO:tasks.workunit.client.0.smithi102.stderr:260487: 3 2024-08-19T15:03:01.608 INFO:tasks.workunit.client.0.smithi102.stderr:260487: 4 2024-08-19T15:03:01.609 INFO:tasks.workunit.client.0.smithi102.stderr:260487: 5 2024-08-19T15:03:01.609 INFO:tasks.workunit.client.0.smithi102.stderr:260487: 6 2024-08-19T15:03:01.609 INFO:tasks.workunit.client.0.smithi102.stderr:260487: 7 2024-08-19T15:03:01.609 INFO:tasks.workunit.client.0.smithi102.stderr:260487: 8 2024-08-19T15:03:01.609 INFO:tasks.workunit.client.0.smithi102.stderr:260487: 9' 2024-08-19T15:03:01.609 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:03:01.609 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:01.609 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:03:01.609 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836578 2024-08-19T15:03:01.609 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836578 2024-08-19T15:03:01.609 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578' 2024-08-19T15:03:01.609 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:01.609 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:03:01.610 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673051 2024-08-19T15:03:01.610 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673051 2024-08-19T15:03:01.610 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673051' 2024-08-19T15:03:01.610 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:02.751 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clon.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:03:02.751 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509527 2024-08-19T15:03:02.751 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509527 2024-08-19T15:03:02.751 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509527' 2024-08-19T15:03:02.751 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:02.751 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:03:02.751 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645337 2024-08-19T15:03:02.752 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645337 2024-08-19T15:03:02.752 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509527 3-309237645337' 2024-08-19T15:03:02.752 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:02.752 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T15:03:02.752 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182469 2024-08-19T15:03:02.752 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182469 2024-08-19T15:03:02.752 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509527 3-309237645337 4-107374182469' 2024-08-19T15:03:02.752 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:02.752 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T15:03:02.752 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598786 2024-08-19T15:03:02.752 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598786 2024-08-19T15:03:02.752 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509527 3-309237645337 4-107374182469 5-446676598786' 2024-08-19T15:03:02.752 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:02.752 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T15:03:02.753 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855420 2024-08-19T15:03:02.753 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855420 2024-08-19T15:03:02.753 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509527 3-309237645337 4-107374182469 5-446676598786 6-150323855420' 2024-08-19T15:03:02.753 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:02.753 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T15:03:02.753 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691889 2024-08-19T15:03:02.753 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691889 2024-08-19T15:03:02.753 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509527 3-309237645337 4-107374182469 5-446676598786 6-150323855420 7-171798691889' 2024-08-19T15:03:02.753 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:02.753 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T15:03:02.981 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cee.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:03:02.981 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509528 2024-08-19T15:03:02.981 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509528 2024-08-19T15:03:02.981 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509528' 2024-08-19T15:03:02.981 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:02.981 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:03:02.981 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645339 2024-08-19T15:03:02.981 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645339 2024-08-19T15:03:02.981 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509528 3-309237645339' 2024-08-19T15:03:02.981 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:02.981 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T15:03:02.982 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182470 2024-08-19T15:03:02.982 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182470 2024-08-19T15:03:02.982 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509528 3-309237645339 4-107374182470' 2024-08-19T15:03:02.982 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:02.982 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T15:03:02.982 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598787 2024-08-19T15:03:02.982 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598787 2024-08-19T15:03:02.982 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509528 3-309237645339 4-107374182470 5-446676598787' 2024-08-19T15:03:02.982 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:02.982 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T15:03:02.982 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855421 2024-08-19T15:03:02.982 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855421 2024-08-19T15:03:02.983 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509528 3-309237645339 4-107374182470 5-446676598787 6-150323855421' 2024-08-19T15:03:02.983 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:02.983 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T15:03:02.983 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691890 2024-08-19T15:03:02.983 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691890 2024-08-19T15:03:02.983 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509528 3-309237645339 4-107374182470 5-446676598787 6-150323855421 7-171798691890' 2024-08-19T15:03:02.983 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:02.983 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T15:03:03.469 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cphtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528364 2024-08-19T15:03:03.469 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528364 2024-08-19T15:03:03.469 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509527 3-309237645337 4-107374182469 5-446676598786 6-150323855420 7-171798691889 8-193273528364' 2024-08-19T15:03:03.469 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:03.469 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T15:03:03.470 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631490 2024-08-19T15:03:03.470 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631490 2024-08-19T15:03:03.470 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509527 3-309237645337 4-107374182469 5-446676598786 6-150323855420 7-171798691889 8-193273528364 9-442381631490' 2024-08-19T15:03:03.470 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:03.470 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836577 2024-08-19T15:03:03.470 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:03.470 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:03:03.470 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836577 2024-08-19T15:03:03.470 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:03.470 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836577 2024-08-19T15:03:03.470 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2024-08-19T15:03:03.470 INFO:tasks.workunit.client.0.smithi102.stderr:260489: waiting osd.0 seq 21474836577 2024-08-19T15:03:03.471 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:03:03.471 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836577 -lt 21474836577 2024-08-19T15:03:03.471 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:03.471 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673050 2024-08-19T15:03:03.471 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:03.471 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:03:03.471 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673050 2024-08-19T15:03:03.471 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:03.471 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673050 2024-08-19T15:03:03.471 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673050' 2024-08-19T15:03:03.471 INFO:tasks.workunit.client.0.smithi102.stderr:260489: waiting osd.1 seq 42949673050 2024-08-19T15:03:03.471 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:03:03.471 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673052 -lt 42949673050 2024-08-19T15:03:03.471 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:03.472 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509527 2024-08-19T15:03:03.472 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:03.472 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:03:03.685 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528365 2024-08-19T15:03:03.685 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528365 2024-08-19T15:03:03.685 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509528 3-309237645339 4-107374182470 5-446676598787 6-150323855421 7-171798691890 8-193273528365' 2024-08-19T15:03:03.685 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:03.685 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T15:03:03.685 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631491 2024-08-19T15:03:03.685 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631491 2024-08-19T15:03:03.685 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509528 3-309237645339 4-107374182470 5-446676598787 6-150323855421 7-171798691890 8-193273528365 9-442381631491' 2024-08-19T15:03:03.685 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:03.686 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836578 2024-08-19T15:03:03.686 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:03.686 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:03:03.686 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836578 2024-08-19T15:03:03.686 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:03.686 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836578 2024-08-19T15:03:03.686 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836578' 2024-08-19T15:03:03.686 INFO:tasks.workunit.client.0.smithi102.stderr:260487: waiting osd.0 seq 21474836578 2024-08-19T15:03:03.686 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:03:03.686 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836578 -lt 21474836578 2024-08-19T15:03:03.686 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:03.686 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673051 2024-08-19T15:03:03.687 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:03.687 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:03:03.687 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673051 2024-08-19T15:03:03.687 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:03.687 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673051 2024-08-19T15:03:03.687 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-08-19T15:03:03.687 INFO:tasks.workunit.client.0.smithi102.stderr:260487: waiting osd.1 seq 42949673051 2024-08-19T15:03:03.687 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:03:03.687 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673052 -lt 42949673051 2024-08-19T15:03:03.687 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:03.687 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509528 2024-08-19T15:03:03.687 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:03.687 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:03:04.356 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/ceptest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509527 2024-08-19T15:03:04.356 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:04.356 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509527 2024-08-19T15:03:04.356 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509527' 2024-08-19T15:03:04.356 INFO:tasks.workunit.client.0.smithi102.stderr:260489: waiting osd.2 seq 64424509527 2024-08-19T15:03:04.356 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:03:04.356 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509528 -lt 64424509527 2024-08-19T15:03:04.356 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:04.356 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645337 2024-08-19T15:03:04.356 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:04.357 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:03:04.357 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645337 2024-08-19T15:03:04.357 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:04.357 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645337 2024-08-19T15:03:04.357 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645337' 2024-08-19T15:03:04.357 INFO:tasks.workunit.client.0.smithi102.stderr:260489: waiting osd.3 seq 309237645337 2024-08-19T15:03:04.357 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:03:04.357 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645339 -lt 309237645337 2024-08-19T15:03:04.357 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:04.357 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182469 2024-08-19T15:03:04.357 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:04.357 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T15:03:04.357 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182469 2024-08-19T15:03:04.357 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:04.358 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182469 2024-08-19T15:03:04.358 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182469' 2024-08-19T15:03:04.358 INFO:tasks.workunit.client.0.smithi102.stderr:260489: waiting osd.4 seq 107374182469 2024-08-19T15:03:04.358 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T15:03:04.358 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182470 -lt 107374182469 2024-08-19T15:03:04.358 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:04.358 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-446676598786 2024-08-19T15:03:04.358 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:04.358 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T15:03:04.358 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-446676598786 2024-08-19T15:03:04.358 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:04.358 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598786 2024-08-19T15:03:04.614 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: htest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509528 2024-08-19T15:03:04.614 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:04.614 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509528 2024-08-19T15:03:04.614 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509528' 2024-08-19T15:03:04.615 INFO:tasks.workunit.client.0.smithi102.stderr:260487: waiting osd.2 seq 64424509528 2024-08-19T15:03:04.615 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:03:04.615 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509528 -lt 64424509528 2024-08-19T15:03:04.615 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:04.615 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645339 2024-08-19T15:03:04.615 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:04.615 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:03:04.615 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645339 2024-08-19T15:03:04.615 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:04.615 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645339 2024-08-19T15:03:04.615 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645339' 2024-08-19T15:03:04.615 INFO:tasks.workunit.client.0.smithi102.stderr:260487: waiting osd.3 seq 309237645339 2024-08-19T15:03:04.615 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:03:04.616 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645339 -lt 309237645339 2024-08-19T15:03:04.616 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:04.616 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182470 2024-08-19T15:03:04.616 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:04.616 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T15:03:04.616 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182470 2024-08-19T15:03:04.616 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:04.616 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182470 2024-08-19T15:03:04.616 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182470' 2024-08-19T15:03:04.616 INFO:tasks.workunit.client.0.smithi102.stderr:260487: waiting osd.4 seq 107374182470 2024-08-19T15:03:04.616 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T15:03:04.616 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182470 -lt 107374182470 2024-08-19T15:03:04.617 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:04.617 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-446676598787 2024-08-19T15:03:04.617 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:04.617 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T15:03:04.617 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-446676598787 2024-08-19T15:03:04.617 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:04.617 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598787 2024-08-19T15:03:05.585 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 446676598786' 2024-08-19T15:03:05.586 INFO:tasks.workunit.client.0.smithi102.stderr:260489: waiting osd.5 seq 446676598786 2024-08-19T15:03:05.586 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T15:03:05.586 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598787 -lt 446676598786 2024-08-19T15:03:05.586 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:05.586 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855420 2024-08-19T15:03:05.586 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:05.586 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T15:03:05.586 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855420 2024-08-19T15:03:05.586 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:05.586 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855420 2024-08-19T15:03:05.586 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855420' 2024-08-19T15:03:05.587 INFO:tasks.workunit.client.0.smithi102.stderr:260489: waiting osd.6 seq 150323855420 2024-08-19T15:03:05.587 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T15:03:05.587 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855421 -lt 150323855420 2024-08-19T15:03:05.587 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:05.587 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691889 2024-08-19T15:03:05.587 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:05.587 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T15:03:05.587 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691889 2024-08-19T15:03:05.587 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:05.587 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691889 2024-08-19T15:03:05.587 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691889' 2024-08-19T15:03:05.587 INFO:tasks.workunit.client.0.smithi102.stderr:260489: waiting osd.7 seq 171798691889 2024-08-19T15:03:05.588 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T15:03:05.588 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691890 -lt 171798691889 2024-08-19T15:03:05.588 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:05.588 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528364 2024-08-19T15:03:05.588 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:05.588 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T15:03:05.588 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528364 2024-08-19T15:03:05.588 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:05.588 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528364 2024-08-19T15:03:05.588 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528364' 2024-08-19T15:03:05.588 INFO:tasks.workunit.client.0.smithi102.stderr:260489: waiting osd.8 seq 193273528364 2024-08-19T15:03:05.588 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T15:03:05.588 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528366 -lt 193273528364 2024-08-19T15:03:05.851 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:22 echo 'waiting osd.5 seq 446676598787' 2024-08-19T15:03:05.851 INFO:tasks.workunit.client.0.smithi102.stderr:260487: waiting osd.5 seq 446676598787 2024-08-19T15:03:05.851 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T15:03:05.851 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598787 -lt 446676598787 2024-08-19T15:03:05.851 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:05.851 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855421 2024-08-19T15:03:05.852 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:05.852 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T15:03:05.852 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855421 2024-08-19T15:03:05.852 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:05.852 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855421 2024-08-19T15:03:05.852 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855421' 2024-08-19T15:03:05.852 INFO:tasks.workunit.client.0.smithi102.stderr:260487: waiting osd.6 seq 150323855421 2024-08-19T15:03:05.852 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T15:03:05.852 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855421 -lt 150323855421 2024-08-19T15:03:05.852 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:05.852 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691890 2024-08-19T15:03:05.852 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:05.852 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T15:03:05.853 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691890 2024-08-19T15:03:05.853 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:05.853 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691890 2024-08-19T15:03:05.853 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691890' 2024-08-19T15:03:05.853 INFO:tasks.workunit.client.0.smithi102.stderr:260487: waiting osd.7 seq 171798691890 2024-08-19T15:03:05.853 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T15:03:05.853 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691890 -lt 171798691890 2024-08-19T15:03:05.853 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:05.853 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528365 2024-08-19T15:03:05.853 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:05.853 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T15:03:05.853 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528365 2024-08-19T15:03:05.853 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:05.854 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528365 2024-08-19T15:03:05.854 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528365' 2024-08-19T15:03:05.854 INFO:tasks.workunit.client.0.smithi102.stderr:260487: waiting osd.8 seq 193273528365 2024-08-19T15:03:05.854 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T15:03:05.854 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528366 -lt 193273528365 2024-08-19T15:03:07.016 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:07.016 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631490 2024-08-19T15:03:07.016 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:07.016 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T15:03:07.016 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631490 2024-08-19T15:03:07.016 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:07.016 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631490 2024-08-19T15:03:07.017 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631490' 2024-08-19T15:03:07.017 INFO:tasks.workunit.client.0.smithi102.stderr:260489: waiting osd.9 seq 442381631490 2024-08-19T15:03:07.017 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T15:03:07.017 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631492 -lt 442381631490 2024-08-19T15:03:07.017 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:03:07.017 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:03:07.017 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:03:07.017 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:03:07.017 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:03:07.017 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:03:07.017 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:03:07.017 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:03:07.017 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:03:07.018 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:03:07.018 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:03:07.018 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:03:07.018 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:03:07.018 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:03:07.018 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:03:07.018 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-08-19T15:03:07.018 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-08-19T15:03:07.018 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T15:03:07.018 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-08-19T15:03:07.018 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T15:03:07.018 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T15:03:07.018 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T15:03:07.019 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:03:07.019 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:03:07.221 INFO:tasks.workunit.client.0.smithi102.stderr:260489210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:07.221 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631491 2024-08-19T15:03:07.221 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:07.221 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T15:03:07.221 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631491 2024-08-19T15:03:07.221 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:07.221 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631491 2024-08-19T15:03:07.221 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631491' 2024-08-19T15:03:07.221 INFO:tasks.workunit.client.0.smithi102.stderr:260487: waiting osd.9 seq 442381631491 2024-08-19T15:03:07.221 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T15:03:07.222 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631492 -lt 442381631491 2024-08-19T15:03:07.222 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:03:07.222 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:03:07.222 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:03:07.222 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:03:07.222 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:03:07.222 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:03:07.222 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:03:07.222 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:03:07.222 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:03:07.222 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:03:07.223 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:03:07.223 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:03:07.223 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:03:07.223 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:03:07.223 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:03:07.223 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-08-19T15:03:07.223 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-08-19T15:03:07.223 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T15:03:07.223 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-08-19T15:03:07.223 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T15:03:07.223 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T15:03:07.223 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T15:03:07.223 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:03:07.224 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:03:07.677 INFO:tasks.workunit.client.0.smithi102.stderr:26048: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:03:07.677 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:03:07.677 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:03:07.677 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:03:07.677 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:03:07.677 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:03:07.677 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:03:07.677 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:03:07.677 INFO:tasks.workunit.client.0.smithi102.stderr:260489: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:03:07.677 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:03:07.677 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:03:07.678 INFO:tasks.workunit.client.0.smithi102.stderr:260489: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:03:07.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T15:03:07.843 INFO:tasks.workunit.client.0.smithi102.stderr:7: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:03:07.843 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:03:07.843 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:03:07.843 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:03:07.843 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:03:07.844 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:03:07.844 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:03:07.844 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:03:07.844 INFO:tasks.workunit.client.0.smithi102.stderr:260487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:03:07.844 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:03:07.844 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:03:07.844 INFO:tasks.workunit.client.0.smithi102.stderr:260487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:03:07.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T15:03:07.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T15:03:07.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 260486 2024-08-19T15:03:07.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-08-19T15:03:07.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-08-19T15:03:07.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-08-19T15:03:07.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-08-19T15:03:07.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-08-19T15:03:07.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T15:03:07.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T15:03:07.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-08-19T15:03:07.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-08-19T15:03:07.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-08-19T15:03:07.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-08-19T15:03:07.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-08-19T15:03:07.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-08-19T15:03:07.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T15:03:07.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T15:03:07.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T15:03:07.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 263815"' 2024-08-19T15:03:07.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 263815' 2024-08-19T15:03:07.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T15:03:07.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T15:03:07.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T15:03:07.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 263816"' 2024-08-19T15:03:07.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 263816' 2024-08-19T15:03:07.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-08-19T15:03:07.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 263815 263816' 2024-08-19T15:03:07.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-08-19T15:03:07.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T15:03:07.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T15:03:07.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 263815 2024-08-19T15:03:07.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/263818: /' 2024-08-19T15:03:07.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T15:03:07.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/263820: /' 2024-08-19T15:03:10.772 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:03:10.772 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:03:10.772 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-08-19T15:03:10.772 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:03:10.772 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T15:03:10.773 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:03:10.773 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:03:10.773 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-08-19T15:03:10.773 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:03:10.773 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-08-19T15:03:10.773 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:03:10.773 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:03:10.773 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:03:10.773 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:03:10.773 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:03:10.773 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:03:10.773 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-08-19T15:03:10.774 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:03:10.774 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:03:10.774 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-08-19T15:03:10.774 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:03:10.774 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-08-19T15:03:10.774 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-08-19T15:03:10.774 INFO:tasks.workunit.client.0.smithi102.stderr:263820: remove 1#2:eb822e21:::SOMETHING:head# 2024-08-19T15:03:10.774 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-08-19T15:03:10.774 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:03:10.774 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:03:10.774 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-08-19T15:03:10.774 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:03:10.774 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-08-19T15:03:10.775 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:03:10.775 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:03:10.776 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:03:10.776 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:03:10.776 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-08-19T15:03:10.776 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-08-19T15:03:10.776 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:03:10.776 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:03:10.776 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:03:10.776 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:03:10.776 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:03:10.776 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:03:10.776 INFO:tasks.workunit.client.0.smithi102.stderr:263820: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:03:10.776 INFO:tasks.workunit.client.0.smithi102.stderr:263820: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:03:10.776 INFO:tasks.workunit.client.0.smithi102.stderr:263820: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:03:10.777 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:03:10.777 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:03:10.777 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:03:10.777 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:03:10.777 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:03:10.777 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:03:10.777 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:03:10.777 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:03:10.777 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:03:10.777 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:03:10.777 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:03:10.777 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-08-19T15:03:10.777 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-08-19T15:03:10.778 INFO:tasks.workunit.client.0.smithi102.stderr:263820: start osd.5 2024-08-19T15:03:10.778 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:03:10.899 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:03:10.899 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:03:10.899 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-08-19T15:03:10.899 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:03:10.899 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T15:03:10.899 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:03:10.899 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:03:10.899 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-08-19T15:03:10.899 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:03:10.899 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-08-19T15:03:10.900 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:03:10.900 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:03:10.900 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:03:10.900 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:03:10.900 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:03:10.900 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:03:10.900 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T15:03:10.900 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:03:10.900 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:03:10.900 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-08-19T15:03:10.900 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:03:10.900 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-08-19T15:03:10.900 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-08-19T15:03:10.901 INFO:tasks.workunit.client.0.smithi102.stderr:263818: remove 0#2:eb822e21:::SOMETHING:head# 2024-08-19T15:03:10.901 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-08-19T15:03:10.901 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:03:10.901 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:03:10.901 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-19T15:03:10.901 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:03:10.901 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T15:03:10.901 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:03:10.901 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:03:10.902 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:03:10.902 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:03:10.902 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T15:03:10.902 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T15:03:10.902 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:03:10.902 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:03:10.902 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:03:10.902 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:03:10.902 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:03:10.903 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:03:10.903 INFO:tasks.workunit.client.0.smithi102.stderr:263818: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:03:10.903 INFO:tasks.workunit.client.0.smithi102.stderr:263818: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:03:10.903 INFO:tasks.workunit.client.0.smithi102.stderr:263818: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:03:10.903 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:03:10.903 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:03:10.903 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:03:10.903 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:03:10.903 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:03:10.903 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:03:10.903 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:03:10.903 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:03:10.903 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:03:10.904 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:03:10.904 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:03:10.904 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T15:03:10.904 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-19T15:03:10.904 INFO:tasks.workunit.client.0.smithi102.stderr:263818: start osd.3 2024-08-19T15:03:10.904 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:03:15.365 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-08-19T15:03:15.366 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-08-19T15:03:15.366 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:03:15.366 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:03:15.366 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:03:15.366 INFO:tasks.workunit.client.0.smithi102.stderr:263820: 2024-08-19T15:03:10.805+0000 7f551c8b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:15.366 INFO:tasks.workunit.client.0.smithi102.stderr:263820: 2024-08-19T15:03:10.808+0000 7f551c8b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:15.366 INFO:tasks.workunit.client.0.smithi102.stderr:263820: 2024-08-19T15:03:10.809+0000 7f551c8b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:15.366 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-08-19T15:03:15.366 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:03:15.366 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-08-19T15:03:15.367 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:03:15.367 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:03:15.367 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:15.367 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:03:15.367 INFO:tasks.workunit.client.0.smithi102.stderr:263820: 0 2024-08-19T15:03:15.367 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:15.367 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:03:15.367 INFO:tasks.workunit.client.0.smithi102.stderr:263820: 2024-08-19T15:03:11.326+0000 7f551c8b1640 -1 Falling back to public interface 2024-08-19T15:03:15.367 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:15.367 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:15.367 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:15.367 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:03:15.368 INFO:tasks.workunit.client.0.smithi102.stderr:263820: 1 2024-08-19T15:03:15.368 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:15.368 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:03:15.368 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:15.368 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:15.368 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:15.368 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:03:15.368 INFO:tasks.workunit.client.0.smithi102.stderr:263820: 2 2024-08-19T15:03:15.368 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:15.368 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:03:15.368 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:15.368 INFO:tasks.workunit.client.0.smithi102.stderr:263820: 2024-08-19T15:03:14.160+0000 7f551c8b1640 -1 osd.5 106 log_to_monitors true 2024-08-19T15:03:15.369 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:15.369 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:15.369 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:03:15.369 INFO:tasks.workunit.client.0.smithi102.stderr:263820: 3 2024-08-19T15:03:15.369 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:15.369 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:03:15.369 INFO:tasks.workunit.client.0.smithi102.stderr:263820: osd.5 up in weight 1 up_from 110 up_thru 110 down_at 107 l263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-08-19T15:03:15.369 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-19T15:03:15.369 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:03:15.369 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:03:15.369 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:03:15.369 INFO:tasks.workunit.client.0.smithi102.stderr:263818: 2024-08-19T15:03:10.928+0000 7f74f45c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:15.369 INFO:tasks.workunit.client.0.smithi102.stderr:263818: 2024-08-19T15:03:10.931+0000 7f74f45c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:15.370 INFO:tasks.workunit.client.0.smithi102.stderr:263818: 2024-08-19T15:03:10.932+0000 7f74f45c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:15.370 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-19T15:03:15.370 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:03:15.370 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T15:03:15.370 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:03:15.370 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:03:15.370 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:15.370 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:03:15.370 INFO:tasks.workunit.client.0.smithi102.stderr:263818: 0 2024-08-19T15:03:15.370 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:15.370 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:03:15.370 INFO:tasks.workunit.client.0.smithi102.stderr:263818: 2024-08-19T15:03:11.446+0000 7f74f45c4640 -1 Falling back to public interface 2024-08-19T15:03:15.370 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:15.371 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:15.371 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:15.371 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:03:15.371 INFO:tasks.workunit.client.0.smithi102.stderr:263818: 1 2024-08-19T15:03:15.371 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:15.371 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:03:15.371 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:15.371 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:15.371 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:15.371 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:03:15.371 INFO:tasks.workunit.client.0.smithi102.stderr:263818: 2 2024-08-19T15:03:15.371 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:15.371 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:03:15.372 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:15.372 INFO:tasks.workunit.client.0.smithi102.stderr:263818: 2024-08-19T15:03:14.261+0000 7f74f45c4640 -1 osd.3 106 log_to_monitors true 2024-08-19T15:03:15.372 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:15.372 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:15.372 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:03:15.372 INFO:tasks.workunit.client.0.smithi102.stderr:263818: 3 2024-08-19T15:03:15.372 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:15.372 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:03:16.237 INFO:tasks.workunit.client.0.smithi102.stderr:263818: osd.3 up in weight 1 up_from 110 up_thru 110 down_at 107 last_clean_interval [72,106) [v2:127.0.0.1:6874/1557570411,v1:127.0.0.1:6875/1557570411] [v2:127.0.0.1:6876/1557570411,v1:127.0.0.1:6877/1557570411] exists,up a53d7b50-dd2b-44ae-862b-772fc9028cb8 2024-08-19T15:03:16.237 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:03:16.237 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:03:16.237 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:03:16.237 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:03:16.237 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:03:16.237 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:03:16.237 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:03:16.237 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:03:16.238 INFO:tasks.workunit.client.0.smithi102.stderr:263818: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:03:16.238 INFO:tasks.workunit.client.0.smithi102.stderr:263818: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:03:16.238 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:03:16.238 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:03:16.238 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:03:16.238 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:03:16.238 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:03:16.238 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:03:16.238 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:03:16.238 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:03:16.238 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:03:16.238 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:03:16.238 INFO:tasks.workunit.client.0.smithi102.stderr:263818: 1 2024-08-19T15:03:16.239 INFO:tasks.workunit.client.0.smithi102.stderr:263818: 2 2024-08-19T15:03:16.239 INFO:tasks.workunit.client.0.smithi102.stderr:263818: 3 2024-08-19T15:03:16.239 INFO:tasks.workunit.client.0.smithi102.stderr:263818: 4 2024-08-19T15:03:16.239 INFO:tasks.workunit.client.0.smithi102.stderr:263818: 5 2024-08-19T15:03:16.239 INFO:tasks.workunit.client.0.smithi102.stderr:263818: 6 2024-08-19T15:03:16.239 INFO:tasks.workunit.client.0.smithi102.stderr:263818: 7 2024-08-19T15:03:16.239 INFO:tasks.workunit.client.0.smithi102.stderr:263818: 8 2024-08-19T15:03:16.239 INFO:tasks.workunit.client.0.smithi102.stderr:263818: 9' 2024-08-19T15:03:16.239 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:03:16.239 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:16.239 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:03:16.239 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836582 2024-08-19T15:03:16.239 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836582 2024-08-19T15:03:16.239 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582' 2024-08-19T15:03:16.240 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:16.240 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:03:16.240 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673055 2024-08-19T15:03:16.240 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673055 2024-08-19T15:03:16.240 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673055' 2024-08-19T15:03:16.240 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:16.240 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.clienast_clean_interval [104,106) [v2:127.0.0.1:6826/1586361457,v1:127.0.0.1:6827/1586361457] [v2:127.0.0.1:6828/1586361457,v1:127.0.0.1:6829/1586361457] exists,up 93853ae7-d7eb-460d-96c3-c71aa1567468 2024-08-19T15:03:16.240 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:03:16.240 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:03:16.241 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:03:16.241 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:03:16.241 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:03:16.241 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:03:16.241 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:03:16.241 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:03:16.241 INFO:tasks.workunit.client.0.smithi102.stderr:263820: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:03:16.241 INFO:tasks.workunit.client.0.smithi102.stderr:263820: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:03:16.241 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:03:16.241 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:03:16.241 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:03:16.241 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:03:16.241 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:03:16.241 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:03:16.242 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:03:16.242 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:03:16.242 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:03:16.242 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:03:16.242 INFO:tasks.workunit.client.0.smithi102.stderr:263820: 1 2024-08-19T15:03:16.242 INFO:tasks.workunit.client.0.smithi102.stderr:263820: 2 2024-08-19T15:03:16.242 INFO:tasks.workunit.client.0.smithi102.stderr:263820: 3 2024-08-19T15:03:16.242 INFO:tasks.workunit.client.0.smithi102.stderr:263820: 4 2024-08-19T15:03:16.242 INFO:tasks.workunit.client.0.smithi102.stderr:263820: 5 2024-08-19T15:03:16.242 INFO:tasks.workunit.client.0.smithi102.stderr:263820: 6 2024-08-19T15:03:16.242 INFO:tasks.workunit.client.0.smithi102.stderr:263820: 7 2024-08-19T15:03:16.242 INFO:tasks.workunit.client.0.smithi102.stderr:263820: 8 2024-08-19T15:03:16.242 INFO:tasks.workunit.client.0.smithi102.stderr:263820: 9' 2024-08-19T15:03:16.243 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:03:16.243 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:16.243 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:03:16.243 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836583 2024-08-19T15:03:16.243 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836583 2024-08-19T15:03:16.243 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583' 2024-08-19T15:03:16.243 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:16.243 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:03:16.243 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673056 2024-08-19T15:03:16.243 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673056 2024-08-19T15:03:16.243 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673056' 2024-08-19T15:03:16.243 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:16.984 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.cliet.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:03:16.984 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509532 2024-08-19T15:03:16.984 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509532 2024-08-19T15:03:16.985 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673055 2-64424509532' 2024-08-19T15:03:16.985 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:16.985 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:03:16.985 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402562 2024-08-19T15:03:16.985 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402562 2024-08-19T15:03:16.985 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673055 2-64424509532 3-472446402562' 2024-08-19T15:03:16.985 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:16.985 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T15:03:16.985 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182474 2024-08-19T15:03:16.985 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182474 2024-08-19T15:03:16.985 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673055 2-64424509532 3-472446402562 4-107374182474' 2024-08-19T15:03:16.985 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:16.985 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T15:03:16.986 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402562 2024-08-19T15:03:16.986 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402562 2024-08-19T15:03:16.986 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673055 2-64424509532 3-472446402562 4-107374182474 5-472446402562' 2024-08-19T15:03:16.986 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:16.986 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T15:03:16.986 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855425 2024-08-19T15:03:16.986 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855425 2024-08-19T15:03:16.986 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673055 2-64424509532 3-472446402562 4-107374182474 5-472446402562 6-150323855425' 2024-08-19T15:03:16.986 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:16.986 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T15:03:16.986 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691894 2024-08-19T15:03:16.987 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691894 2024-08-19T15:03:16.987 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673055 2-64424509532 3-472446402562 4-107374182474 5-472446402562 6-150323855425 7-171798691894' 2024-08-19T15:03:16.987 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:16.987 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T15:03:16.987 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtestnt.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:03:16.987 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509533 2024-08-19T15:03:16.987 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509533 2024-08-19T15:03:16.987 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673056 2-64424509533' 2024-08-19T15:03:16.987 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:16.987 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:03:16.987 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402563 2024-08-19T15:03:16.987 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402563 2024-08-19T15:03:16.988 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673056 2-64424509533 3-472446402563' 2024-08-19T15:03:16.988 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:16.988 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T15:03:16.988 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182475 2024-08-19T15:03:16.988 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182475 2024-08-19T15:03:16.988 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673056 2-64424509533 3-472446402563 4-107374182475' 2024-08-19T15:03:16.988 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:16.988 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T15:03:16.988 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402563 2024-08-19T15:03:16.988 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402563 2024-08-19T15:03:16.988 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673056 2-64424509533 3-472446402563 4-107374182475 5-472446402563' 2024-08-19T15:03:16.988 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:16.988 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T15:03:16.989 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855424 2024-08-19T15:03:16.989 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855424 2024-08-19T15:03:16.989 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673056 2-64424509533 3-472446402563 4-107374182475 5-472446402563 6-150323855424' 2024-08-19T15:03:16.989 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:16.989 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T15:03:16.989 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691895 2024-08-19T15:03:16.989 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691895 2024-08-19T15:03:16.989 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673056 2-64424509533 3-472446402563 4-107374182475 5-472446402563 6-150323855424 7-171798691895' 2024-08-19T15:03:16.989 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:16.989 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T15:03:17.655 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528370 2024-08-19T15:03:17.655 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528370 2024-08-19T15:03:17.655 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673056 2-64424509533 3-472446402563 4-107374182475 5-472446402563 6-150323855424 7-171798691895 8-193273528370' 2024-08-19T15:03:17.656 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:17.656 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T15:03:17.656 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631495 2024-08-19T15:03:17.656 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631495 2024-08-19T15:03:17.656 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673056 2-64424509533 3-472446402563 4-107374182475 5-472446402563 6-150323855424 7-171798691895 8-193273528370 9-442381631495' 2024-08-19T15:03:17.656 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:17.656 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836583 2024-08-19T15:03:17.656 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:17.656 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:03:17.656 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836583 2024-08-19T15:03:17.656 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:17.657 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836583 2024-08-19T15:03:17.657 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836583' 2024-08-19T15:03:17.657 INFO:tasks.workunit.client.0.smithi102.stderr:263820: waiting osd.0 seq 21474836583 2024-08-19T15:03:17.657 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:03:17.657 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836583 -lt 21474836583 2024-08-19T15:03:17.657 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:17.657 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673056 2024-08-19T15:03:17.657 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:17.657 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:03:17.657 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673056 2024-08-19T15:03:17.657 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:17.657 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673056 2024-08-19T15:03:17.657 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2024-08-19T15:03:17.658 INFO:tasks.workunit.client.0.smithi102.stderr:263820: waiting osd.1 seq 42949673056 2024-08-19T15:03:17.658 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:03:17.658 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673057 -lt 42949673056 2024-08-19T15:03:17.658 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:17.658 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509533 2024-08-19T15:03:17.658 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:17.658 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:03:17.664 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest//clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528369 2024-08-19T15:03:17.664 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528369 2024-08-19T15:03:17.664 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673055 2-64424509532 3-472446402562 4-107374182474 5-472446402562 6-150323855425 7-171798691894 8-193273528369' 2024-08-19T15:03:17.664 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:17.664 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T15:03:17.664 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631496 2024-08-19T15:03:17.664 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631496 2024-08-19T15:03:17.664 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673055 2-64424509532 3-472446402562 4-107374182474 5-472446402562 6-150323855425 7-171798691894 8-193273528369 9-442381631496' 2024-08-19T15:03:17.664 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:17.664 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836582 2024-08-19T15:03:17.665 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:17.665 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:03:17.665 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836582 2024-08-19T15:03:17.665 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:17.665 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836582 2024-08-19T15:03:17.665 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2024-08-19T15:03:17.665 INFO:tasks.workunit.client.0.smithi102.stderr:263818: waiting osd.0 seq 21474836582 2024-08-19T15:03:17.665 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:03:17.665 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836583 -lt 21474836582 2024-08-19T15:03:17.665 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:17.665 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673055 2024-08-19T15:03:17.665 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:17.666 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:03:17.666 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673055 2024-08-19T15:03:17.666 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:17.666 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673055 2024-08-19T15:03:17.666 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-08-19T15:03:17.666 INFO:tasks.workunit.client.0.smithi102.stderr:263818: waiting osd.1 seq 42949673055 2024-08-19T15:03:17.666 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:03:17.666 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673057 -lt 42949673055 2024-08-19T15:03:17.666 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:17.666 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509532 2024-08-19T15:03:17.666 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:17.666 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:03:18.530 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/cclone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509533 2024-08-19T15:03:18.530 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:18.530 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509533 2024-08-19T15:03:18.530 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509533' 2024-08-19T15:03:18.530 INFO:tasks.workunit.client.0.smithi102.stderr:263820: waiting osd.2 seq 64424509533 2024-08-19T15:03:18.531 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:03:18.531 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509533 -lt 64424509533 2024-08-19T15:03:18.531 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:18.531 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-472446402563 2024-08-19T15:03:18.531 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:18.531 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:03:18.531 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-472446402563 2024-08-19T15:03:18.531 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:18.531 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402563 2024-08-19T15:03:18.531 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 472446402563' 2024-08-19T15:03:18.531 INFO:tasks.workunit.client.0.smithi102.stderr:263820: waiting osd.3 seq 472446402563 2024-08-19T15:03:18.531 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:03:18.532 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402563 -lt 472446402563 2024-08-19T15:03:18.532 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:18.532 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182475 2024-08-19T15:03:18.532 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:18.532 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T15:03:18.532 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182475 2024-08-19T15:03:18.532 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:18.532 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182475 2024-08-19T15:03:18.532 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182475' 2024-08-19T15:03:18.532 INFO:tasks.workunit.client.0.smithi102.stderr:263820: waiting osd.4 seq 107374182475 2024-08-19T15:03:18.532 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T15:03:18.532 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182475 -lt 107374182475 2024-08-19T15:03:18.532 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:18.533 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-472446402563 2024-08-19T15:03:18.533 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:18.533 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T15:03:18.533 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-472446402563 2024-08-19T15:03:18.533 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:18.533 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402563 2024-08-19T15:03:18.610 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echolone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509532 2024-08-19T15:03:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509532 2024-08-19T15:03:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-08-19T15:03:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:263818: waiting osd.2 seq 64424509532 2024-08-19T15:03:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:03:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509533 -lt 64424509532 2024-08-19T15:03:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-472446402562 2024-08-19T15:03:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:03:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-472446402562 2024-08-19T15:03:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402562 2024-08-19T15:03:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 472446402562' 2024-08-19T15:03:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:263818: waiting osd.3 seq 472446402562 2024-08-19T15:03:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:03:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402563 -lt 472446402562 2024-08-19T15:03:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182474 2024-08-19T15:03:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:18.612 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T15:03:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182474 2024-08-19T15:03:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182474 2024-08-19T15:03:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182474' 2024-08-19T15:03:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:263818: waiting osd.4 seq 107374182474 2024-08-19T15:03:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T15:03:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182475 -lt 107374182474 2024-08-19T15:03:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-472446402562 2024-08-19T15:03:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T15:03:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-472446402562 2024-08-19T15:03:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:18.614 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402562 2024-08-19T15:03:19.762 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 472446402562' 2024-08-19T15:03:19.762 INFO:tasks.workunit.client.0.smithi102.stderr:263818: waiting osd.5 seq 472446402562 2024-08-19T15:03:19.762 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T15:03:19.762 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402563 -lt 472446402562 2024-08-19T15:03:19.762 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:19.762 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855425 2024-08-19T15:03:19.763 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:19.763 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T15:03:19.763 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855425 2024-08-19T15:03:19.763 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:19.763 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855425 2024-08-19T15:03:19.763 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855425' 2024-08-19T15:03:19.764 INFO:tasks.workunit.client.0.smithi102.stderr:263818: waiting osd.6 seq 150323855425 2024-08-19T15:03:19.764 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T15:03:19.764 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855426 -lt 150323855425 2024-08-19T15:03:19.764 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:19.764 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691894 2024-08-19T15:03:19.765 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:19.765 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T15:03:19.765 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691894 2024-08-19T15:03:19.765 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:19.765 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691894 2024-08-19T15:03:19.765 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691894' 2024-08-19T15:03:19.766 INFO:tasks.workunit.client.0.smithi102.stderr:263818: waiting osd.7 seq 171798691894 2024-08-19T15:03:19.766 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T15:03:19.766 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691895 -lt 171798691894 2024-08-19T15:03:19.766 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:19.766 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528369 2024-08-19T15:03:19.766 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:19.766 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T15:03:19.767 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528369 2024-08-19T15:03:19.767 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:19.767 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528369 2024-08-19T15:03:19.767 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2024-08-19T15:03:19.767 INFO:tasks.workunit.client.0.smithi102.stderr:263818: waiting osd.8 seq 193273528369 2024-08-19T15:03:19.767 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T15:03:19.767 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528371 -lt 193273528369 2024-08-19T15:03:19.769 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: fl 'waiting osd.5 seq 472446402563' 2024-08-19T15:03:19.770 INFO:tasks.workunit.client.0.smithi102.stderr:263820: waiting osd.5 seq 472446402563 2024-08-19T15:03:19.770 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T15:03:19.770 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402563 -lt 472446402563 2024-08-19T15:03:19.770 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:19.770 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855424 2024-08-19T15:03:19.770 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:19.770 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T15:03:19.770 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855424 2024-08-19T15:03:19.770 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:19.770 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855424 2024-08-19T15:03:19.771 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855424' 2024-08-19T15:03:19.771 INFO:tasks.workunit.client.0.smithi102.stderr:263820: waiting osd.6 seq 150323855424 2024-08-19T15:03:19.771 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T15:03:19.771 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855426 -lt 150323855424 2024-08-19T15:03:19.771 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:19.771 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691895 2024-08-19T15:03:19.771 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:19.771 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T15:03:19.771 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691895 2024-08-19T15:03:19.771 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:19.771 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691895 2024-08-19T15:03:19.771 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691895' 2024-08-19T15:03:19.772 INFO:tasks.workunit.client.0.smithi102.stderr:263820: waiting osd.7 seq 171798691895 2024-08-19T15:03:19.772 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T15:03:19.772 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691895 -lt 171798691895 2024-08-19T15:03:19.772 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:19.772 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528370 2024-08-19T15:03:19.772 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:19.772 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T15:03:19.772 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528370 2024-08-19T15:03:19.772 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:19.772 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528370 2024-08-19T15:03:19.772 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528370' 2024-08-19T15:03:19.773 INFO:tasks.workunit.client.0.smithi102.stderr:263820: waiting osd.8 seq 193273528370 2024-08-19T15:03:19.773 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T15:03:19.773 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528371 -lt 193273528370 2024-08-19T15:03:21.027 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: fush_pg_stats: for s in $seqs 2024-08-19T15:03:21.027 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631496 2024-08-19T15:03:21.027 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:21.027 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T15:03:21.027 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631496 2024-08-19T15:03:21.027 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:21.027 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631496 2024-08-19T15:03:21.027 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631496' 2024-08-19T15:03:21.028 INFO:tasks.workunit.client.0.smithi102.stderr:263818: waiting osd.9 seq 442381631496 2024-08-19T15:03:21.028 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T15:03:21.028 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631496 -lt 442381631496 2024-08-19T15:03:21.028 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:03:21.028 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:03:21.028 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:03:21.028 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:03:21.028 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:03:21.028 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:03:21.028 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:03:21.028 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:03:21.029 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:03:21.029 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:03:21.029 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:03:21.029 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:03:21.029 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:03:21.029 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:03:21.029 INFO:tasks.workunit.client.0.smithi102.stderr:263818: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:03:21.029 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:03:21.029 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:03:21.029 INFO:tasks.workunit.client.0.smithi102.stderr:263818: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:03:21.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T15:03:21.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T15:03:21.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 263816 2024-08-19T15:03:21.113 INFO:tasks.workunit.client.0.smithi102.stderr:lush_pg_stats: for s in $seqs 2024-08-19T15:03:21.114 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631495 2024-08-19T15:03:21.114 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:21.114 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T15:03:21.114 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631495 2024-08-19T15:03:21.114 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:21.114 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631495 2024-08-19T15:03:21.114 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631495' 2024-08-19T15:03:21.114 INFO:tasks.workunit.client.0.smithi102.stderr:263820: waiting osd.9 seq 442381631495 2024-08-19T15:03:21.114 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T15:03:21.114 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631496 -lt 442381631495 2024-08-19T15:03:21.115 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:03:21.115 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:03:21.115 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:03:21.115 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:03:21.115 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:03:21.115 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:03:21.115 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:03:21.115 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:03:21.115 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:03:21.115 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:03:21.115 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:03:21.115 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:03:21.116 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:03:21.116 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:03:21.116 INFO:tasks.workunit.client.0.smithi102.stderr:263820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:03:21.116 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:03:21.116 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:03:21.116 INFO:tasks.workunit.client.0.smithi102.stderr:263820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:03:21.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T15:03:21.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-08-19T15:03:21.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-08-19T15:03:21.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-08-19T15:03:21.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-08-19T15:03:21.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-08-19T15:03:21.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-08-19T15:03:21.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T15:03:21.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T15:03:21.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T15:03:21.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:03:21.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-08-19T15:03:21.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-08-19T15:03:21.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-08-19T15:03:21.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-08-19T15:03:21.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:03:21.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:03:21.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:03:21.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:03:21.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T15:02:47.305328+0000 2024-08-19T15:03:21.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-08-19T15:03:21.904 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-08-19T15:03:21.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-08-19T15:02:47.305328+0000 2024-08-19T15:03:21.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T15:03:21.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:02:47.305328+0000 2024-08-19T15:03:21.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:03:21.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:03:21.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:03:21.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:03:21.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:03:21.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:03:21.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:03:21.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:03:22.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:02:47.305328+0000 '>' 2024-08-19T15:02:47.305328+0000 2024-08-19T15:03:22.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:03:23.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:03:23.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:03:23.182 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:03:23.182 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:03:23.182 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:03:23.182 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:03:23.182 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:03:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:02:47.305328+0000 '>' 2024-08-19T15:02:47.305328+0000 2024-08-19T15:03:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:03:24.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:03:24.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:03:24.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:03:24.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:03:24.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:03:24.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:03:24.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:03:24.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:02:47.305328+0000 '>' 2024-08-19T15:02:47.305328+0000 2024-08-19T15:03:24.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:03:25.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:03:25.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:03:25.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:03:25.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:03:25.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:03:25.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:03:25.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:03:26.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:03:22.412057+0000 '>' 2024-08-19T15:02:47.305328+0000 2024-08-19T15:03:26.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:03:26.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-08-19T15:03:26.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T15:03:26.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T15:03:26.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T15:03:26.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 267196"' 2024-08-19T15:03:26.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 267196' 2024-08-19T15:03:26.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-08-19T15:03:26.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T15:03:26.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T15:03:26.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 267198"' 2024-08-19T15:03:26.013 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 267198' 2024-08-19T15:03:26.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T15:03:26.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-08-19T15:03:26.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 267196 267198' 2024-08-19T15:03:26.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-08-19T15:03:26.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T15:03:26.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 267196 2024-08-19T15:03:26.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/267199: /' 2024-08-19T15:03:26.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-08-19T15:03:26.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/267201: /' 2024-08-19T15:03:29.022 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:03:29.023 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:03:29.023 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-08-19T15:03:29.023 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:03:29.023 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T15:03:29.023 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:03:29.023 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:03:29.023 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-08-19T15:03:29.023 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:03:29.023 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-08-19T15:03:29.023 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:03:29.023 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:03:29.024 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:03:29.024 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:03:29.024 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:03:29.024 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:03:29.024 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-08-19T15:03:29.024 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:03:29.024 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:03:29.024 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-08-19T15:03:29.024 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:03:29.024 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-08-19T15:03:29.024 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-08-19T15:03:29.024 INFO:tasks.workunit.client.0.smithi102.stderr:267199: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-08-19T15:03:29.024 INFO:tasks.workunit.client.0.smithi102.stderr:267199: _ 2024-08-19T15:03:29.025 INFO:tasks.workunit.client.0.smithi102.stderr:267199: hinfo_key 2024-08-19T15:03:29.025 INFO:tasks.workunit.client.0.smithi102.stderr:267199: snapset 2024-08-19T15:03:29.025 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-08-19T15:03:29.025 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:03:29.025 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:03:29.025 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-19T15:03:29.025 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:03:29.025 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T15:03:29.025 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:03:29.026 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:03:29.026 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:03:29.026 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:03:29.026 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T15:03:29.026 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T15:03:29.026 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:03:29.026 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:03:29.026 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:03:29.026 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:03:29.026 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:03:29.026 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:03:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:267199: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:03:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:267199: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:03:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:267199: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:03:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:03:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:03:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:03:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:03:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:03:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:03:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:03:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:03:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:03:29.028 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:03:29.028 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:03:29.028 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T15:03:29.028 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-19T15:03:29.028 INFO:tasks.workunit.client.0.smithi102.stderr:267199: start osd.3 2024-08-19T15:03:29.312 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:03:29.312 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:03:29.312 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-08-19T15:03:29.313 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:03:29.313 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-08-19T15:03:29.313 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:03:29.313 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:03:29.313 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-08-19T15:03:29.313 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:03:29.313 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-08-19T15:03:29.313 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:03:29.313 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:03:29.313 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:03:29.313 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:03:29.313 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:03:29.313 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:03:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-08-19T15:03:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:03:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:03:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-08-19T15:03:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:03:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-08-19T15:03:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-08-19T15:03:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:267201: _ 2024-08-19T15:03:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:267201: hinfo_key 2024-08-19T15:03:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:267201: snapset 2024-08-19T15:03:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-08-19T15:03:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:03:29.315 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:03:29.315 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-08-19T15:03:29.315 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:03:29.315 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-08-19T15:03:29.315 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:03:29.315 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:03:29.315 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:03:29.315 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:03:29.315 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-08-19T15:03:29.315 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-08-19T15:03:29.316 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:03:29.316 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:03:29.316 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:03:29.316 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:03:29.316 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:03:29.316 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:03:29.316 INFO:tasks.workunit.client.0.smithi102.stderr:267201: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:03:29.316 INFO:tasks.workunit.client.0.smithi102.stderr:267201: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:03:29.316 INFO:tasks.workunit.client.0.smithi102.stderr:267201: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:03:29.316 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:03:29.316 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:03:29.316 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:03:29.316 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:03:29.317 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:03:29.317 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:03:29.317 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:03:29.317 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:03:29.317 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:03:29.317 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:03:29.317 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:03:29.317 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-08-19T15:03:29.317 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-08-19T15:03:29.317 INFO:tasks.workunit.client.0.smithi102.stderr:267201: start osd.5 2024-08-19T15:03:33.176 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:03:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-08-19T15:03:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-19T15:03:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:03:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:03:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:03:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:267199: 2024-08-19T15:03:29.055+0000 7f73d3863640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:267199: 2024-08-19T15:03:29.057+0000 7f73d3863640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:267199: 2024-08-19T15:03:29.058+0000 7f73d3863640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-19T15:03:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:03:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T15:03:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:03:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:03:33.178 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:33.178 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:03:33.178 INFO:tasks.workunit.client.0.smithi102.stderr:267199: 0 2024-08-19T15:03:33.178 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:33.178 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:03:33.178 INFO:tasks.workunit.client.0.smithi102.stderr:267199: 2024-08-19T15:03:29.573+0000 7f73d3863640 -1 Falling back to public interface 2024-08-19T15:03:33.178 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:33.178 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:33.178 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:33.178 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:03:33.178 INFO:tasks.workunit.client.0.smithi102.stderr:267199: 1 2024-08-19T15:03:33.178 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:33.179 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:03:33.179 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:33.179 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:33.179 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:33.179 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:03:33.179 INFO:tasks.workunit.client.0.smithi102.stderr:267199: 2 2024-08-19T15:03:33.179 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:33.179 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:03:33.179 INFO:tasks.workunit.client.0.smithi102.stderr:267199: 2024-08-19T15:03:32.041+0000 7f73d3863640 -1 osd.3 111 log_to_monitors true 2024-08-19T15:03:33.179 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:33.179 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:33.179 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:33.179 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:03:33.179 INFO:tasks.workunit.client.0.smithi102.stderr:267199: 3 2024-08-19T15:03:33.180 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:33.964 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-08-19T15:03:33.964 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-08-19T15:03:33.964 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-08-19T15:03:33.965 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:03:33.965 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:03:33.965 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:03:33.965 INFO:tasks.workunit.client.0.smithi102.stderr:267201: 2024-08-19T15:03:29.341+0000 7f96e2b55640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:33.965 INFO:tasks.workunit.client.0.smithi102.stderr:267201: 2024-08-19T15:03:29.343+0000 7f96e2b55640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:33.965 INFO:tasks.workunit.client.0.smithi102.stderr:267201: 2024-08-19T15:03:29.344+0000 7f96e2b55640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:33.965 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-08-19T15:03:33.965 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:03:33.965 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-08-19T15:03:33.965 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:03:33.965 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:03:33.965 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:33.965 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:03:33.966 INFO:tasks.workunit.client.0.smithi102.stderr:267201: 0 2024-08-19T15:03:33.966 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:33.966 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:03:33.966 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:33.966 INFO:tasks.workunit.client.0.smithi102.stderr:267201: 2024-08-19T15:03:29.863+0000 7f96e2b55640 -1 Falling back to public interface 2024-08-19T15:03:33.966 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:33.966 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:33.966 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:03:33.966 INFO:tasks.workunit.client.0.smithi102.stderr:267201: 1 2024-08-19T15:03:33.966 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:33.966 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:03:33.966 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:33.966 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:33.967 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:33.967 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:03:33.967 INFO:tasks.workunit.client.0.smithi102.stderr:267201: 2 2024-08-19T15:03:33.967 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:33.967 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:03:33.967 INFO:tasks.workunit.client.0.smithi102.stderr:267201: 2024-08-19T15:03:32.608+0000 7f96e2b55640 -1 osd.5 111 log_to_monitors true 2024-08-19T15:03:33.967 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:33.967 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:33.967 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:33.967 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:03:33.967 INFO:tasks.workunit.client.0.smithi102.stderr:267201: 3 2024-08-19T15:03:33.967 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:33.967 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:03:34.526 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/rs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:03:34.526 INFO:tasks.workunit.client.0.smithi102.stderr:267199: osd.3 up in weight 1 up_from 116 up_thru 110 down_at 112 last_clean_interval [110,111) [v2:127.0.0.1:6826/2524627460,v1:127.0.0.1:6827/2524627460] [v2:127.0.0.1:6828/2524627460,v1:127.0.0.1:6829/2524627460] exists,up a53d7b50-dd2b-44ae-862b-772fc9028cb8 2024-08-19T15:03:34.526 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:03:34.526 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:03:34.526 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:03:34.526 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:03:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:03:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:03:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:03:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:03:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:267199: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:03:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:267199: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:03:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:03:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:03:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:03:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:03:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:03:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:03:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:03:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:03:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:03:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:03:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:267199: 1 2024-08-19T15:03:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:267199: 2 2024-08-19T15:03:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:267199: 3 2024-08-19T15:03:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:267199: 4 2024-08-19T15:03:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:267199: 5 2024-08-19T15:03:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:267199: 6 2024-08-19T15:03:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:267199: 7 2024-08-19T15:03:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:267199: 8 2024-08-19T15:03:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:267199: 9' 2024-08-19T15:03:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:03:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:03:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836587 2024-08-19T15:03:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836587 2024-08-19T15:03:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587' 2024-08-19T15:03:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:03:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673061 2024-08-19T15:03:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673061 2024-08-19T15:03:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673061' 2024-08-19T15:03:35.936 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:35.936 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:03:35.937 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509537 2024-08-19T15:03:35.937 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509537 2024-08-19T15:03:35.937 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673061 2-64424509537' 2024-08-19T15:03:35.937 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:35.937 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:03:35.937 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206338 2024-08-19T15:03:35.937 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206338 2024-08-19T15:03:35.937 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673061 2-64424509537 3-498216206338' 2024-08-19T15:03:35.937 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:35.937 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T15:03:35.937 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182480 2024-08-19T15:03:35.937 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182480 2024-08-19T15:03:35.937 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673061 2-64424509537 3-498216206338 4-107374182480' 2024-08-19T15:03:35.938 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:35.938 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T15:03:35.938 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=511101108226 2024-08-19T15:03:35.938 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 511101108226 2024-08-19T15:03:35.938 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673061 2-64424509537 3-498216206338 4-107374182480 5-511101108226' 2024-08-19T15:03:35.938 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:35.938 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T15:03:35.938 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855430 2024-08-19T15:03:35.938 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855430 2024-08-19T15:03:35.938 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673061 2-64424509537 3-498216206338 4-107374182480 5-511101108226 6-150323855430' 2024-08-19T15:03:35.938 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:35.938 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T15:03:35.939 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691900 2024-08-19T15:03:35.939 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691900 2024-08-19T15:03:35.939 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673061 2-64424509537 3-498216206338 4-107374182480 5-511101108226 6-150323855430 7-171798691900' 2024-08-19T15:03:35.939 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:36.161 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:36.162 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:36.162 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:36.162 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:03:36.162 INFO:tasks.workunit.client.0.smithi102.stderr:267201: 4 2024-08-19T15:03:36.162 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:36.162 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-08-19T15:03:36.162 INFO:tasks.workunit.client.0.smithi102.stderr:267201: osd.5 up in weight 1 up_from 119 up_thru 119 down_at 112 last_clean_interval [110,111) [v2:127.0.0.1:6874/1070782199,v1:127.0.0.1:6875/1070782199] [v2:127.0.0.1:6876/1070782199,v1:127.0.0.1:6877/1070782199] exists,up 93853ae7-d7eb-460d-96c3-c71aa1567468 2024-08-19T15:03:36.162 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:03:36.162 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:03:36.162 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:03:36.163 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:03:36.163 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:03:36.163 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:03:36.163 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:03:36.163 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:03:36.163 INFO:tasks.workunit.client.0.smithi102.stderr:267201: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:03:36.163 INFO:tasks.workunit.client.0.smithi102.stderr:267201: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:03:36.163 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:03:36.163 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:03:36.163 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:03:36.163 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:03:36.163 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:03:36.163 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:03:36.164 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:03:36.164 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:03:36.164 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:03:36.164 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:03:36.164 INFO:tasks.workunit.client.0.smithi102.stderr:267201: 1 2024-08-19T15:03:36.164 INFO:tasks.workunit.client.0.smithi102.stderr:267201: 2 2024-08-19T15:03:36.164 INFO:tasks.workunit.client.0.smithi102.stderr:267201: 3 2024-08-19T15:03:36.164 INFO:tasks.workunit.client.0.smithi102.stderr:267201: 4 2024-08-19T15:03:36.164 INFO:tasks.workunit.client.0.smithi102.stderr:267201: 5 2024-08-19T15:03:36.164 INFO:tasks.workunit.client.0.smithi102.stderr:267201: 6 2024-08-19T15:03:36.164 INFO:tasks.workunit.client.0.smithi102.stderr:267201: 7 2024-08-19T15:03:36.164 INFO:tasks.workunit.client.0.smithi102.stderr:267201: 8 2024-08-19T15:03:36.164 INFO:tasks.workunit.client.0.smithi102.stderr:267201: 9' 2024-08-19T15:03:36.165 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:03:36.165 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:36.165 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:03:36.165 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836589 2024-08-19T15:03:36.165 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836589 2024-08-19T15:03:36.165 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589' 2024-08-19T15:03:36.759 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd a/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T15:03:36.759 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528375 2024-08-19T15:03:36.759 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528375 2024-08-19T15:03:36.759 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673061 2-64424509537 3-498216206338 4-107374182480 5-511101108226 6-150323855430 7-171798691900 8-193273528375' 2024-08-19T15:03:36.759 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:36.759 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T15:03:36.760 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631501 2024-08-19T15:03:36.760 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631501 2024-08-19T15:03:36.760 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673061 2-64424509537 3-498216206338 4-107374182480 5-511101108226 6-150323855430 7-171798691900 8-193273528375 9-442381631501' 2024-08-19T15:03:36.760 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:36.760 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836587 2024-08-19T15:03:36.760 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:36.760 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:03:36.760 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836587 2024-08-19T15:03:36.761 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:36.761 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836587 2024-08-19T15:03:36.761 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2024-08-19T15:03:36.761 INFO:tasks.workunit.client.0.smithi102.stderr:267199: waiting osd.0 seq 21474836587 2024-08-19T15:03:36.761 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:03:36.761 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836588 -lt 21474836587 2024-08-19T15:03:36.761 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:36.761 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673061 2024-08-19T15:03:36.761 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:36.761 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:03:36.761 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673061 2024-08-19T15:03:36.761 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:36.761 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673061 2024-08-19T15:03:36.761 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673061' 2024-08-19T15:03:36.762 INFO:tasks.workunit.client.0.smithi102.stderr:267199: waiting osd.1 seq 42949673061 2024-08-19T15:03:36.762 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:03:36.762 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673061 -lt 42949673061 2024-08-19T15:03:36.762 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:36.762 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509537 2024-08-19T15:03:36.762 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:36.958 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cepin $ids 2024-08-19T15:03:36.958 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:03:36.958 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673062 2024-08-19T15:03:36.958 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673062 2024-08-19T15:03:36.958 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673062' 2024-08-19T15:03:36.958 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:36.958 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:03:36.959 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509539 2024-08-19T15:03:36.959 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509539 2024-08-19T15:03:36.959 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673062 2-64424509539' 2024-08-19T15:03:36.959 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:36.959 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:03:36.959 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206339 2024-08-19T15:03:36.959 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206339 2024-08-19T15:03:36.959 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673062 2-64424509539 3-498216206339' 2024-08-19T15:03:36.959 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:36.959 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-08-19T15:03:36.960 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182481 2024-08-19T15:03:36.960 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182481 2024-08-19T15:03:36.960 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673062 2-64424509539 3-498216206339 4-107374182481' 2024-08-19T15:03:36.960 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:36.960 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-08-19T15:03:36.960 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=511101108227 2024-08-19T15:03:36.960 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 511101108227 2024-08-19T15:03:36.960 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673062 2-64424509539 3-498216206339 4-107374182481 5-511101108227' 2024-08-19T15:03:36.960 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:36.960 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-08-19T15:03:36.960 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855431 2024-08-19T15:03:36.960 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855431 2024-08-19T15:03:36.961 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673062 2-64424509539 3-498216206339 4-107374182481 5-511101108227 6-150323855431' 2024-08-19T15:03:36.961 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:36.961 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-08-19T15:03:37.549 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691901 2024-08-19T15:03:37.549 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691901 2024-08-19T15:03:37.549 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673062 2-64424509539 3-498216206339 4-107374182481 5-511101108227 6-150323855431 7-171798691901' 2024-08-19T15:03:37.550 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:37.550 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-08-19T15:03:37.550 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528376 2024-08-19T15:03:37.550 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528376 2024-08-19T15:03:37.550 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673062 2-64424509539 3-498216206339 4-107374182481 5-511101108227 6-150323855431 7-171798691901 8-193273528376' 2024-08-19T15:03:37.550 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:03:37.550 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-08-19T15:03:37.550 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631502 2024-08-19T15:03:37.550 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631502 2024-08-19T15:03:37.551 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673062 2-64424509539 3-498216206339 4-107374182481 5-511101108227 6-150323855431 7-171798691901 8-193273528376 9-442381631502' 2024-08-19T15:03:37.551 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:37.551 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836589 2024-08-19T15:03:37.551 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:37.551 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:03:37.551 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836589 2024-08-19T15:03:37.551 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:37.551 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836589 2024-08-19T15:03:37.551 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2024-08-19T15:03:37.551 INFO:tasks.workunit.client.0.smithi102.stderr:267201: waiting osd.0 seq 21474836589 2024-08-19T15:03:37.551 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:03:37.552 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836589 -lt 21474836589 2024-08-19T15:03:37.552 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:37.552 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673062 2024-08-19T15:03:37.552 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:37.552 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:03:37.552 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673062 2024-08-19T15:03:37.552 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:37.552 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673062 2024-08-19T15:03:37.552 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673062' 2024-08-19T15:03:37.552 INFO:tasks.workunit.client.0.smithi102.stderr:267201: waiting osd.1 seq 42949673062 2024-08-19T15:03:37.741 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shhtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:03:37.741 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509537 2024-08-19T15:03:37.741 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:37.741 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509537 2024-08-19T15:03:37.741 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2024-08-19T15:03:37.741 INFO:tasks.workunit.client.0.smithi102.stderr:267199: waiting osd.2 seq 64424509537 2024-08-19T15:03:37.741 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:03:37.741 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509537 -lt 64424509537 2024-08-19T15:03:37.741 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:37.742 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-498216206338 2024-08-19T15:03:37.742 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:37.742 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:03:37.742 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-498216206338 2024-08-19T15:03:37.742 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:37.742 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206338 2024-08-19T15:03:37.742 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 498216206338' 2024-08-19T15:03:37.742 INFO:tasks.workunit.client.0.smithi102.stderr:267199: waiting osd.3 seq 498216206338 2024-08-19T15:03:37.742 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:03:37.742 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206339 -lt 498216206338 2024-08-19T15:03:37.742 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:37.742 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182480 2024-08-19T15:03:37.743 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:37.743 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T15:03:37.743 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182480 2024-08-19T15:03:37.743 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:37.743 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182480 2024-08-19T15:03:37.743 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182480' 2024-08-19T15:03:37.743 INFO:tasks.workunit.client.0.smithi102.stderr:267199: waiting osd.4 seq 107374182480 2024-08-19T15:03:37.743 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T15:03:37.743 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182481 -lt 107374182480 2024-08-19T15:03:37.743 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:37.743 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-511101108226 2024-08-19T15:03:37.743 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:37.743 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T15:03:37.744 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-511101108226 2024-08-19T15:03:37.744 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:38.799 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=5111:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:03:38.799 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673063 -lt 42949673062 2024-08-19T15:03:38.800 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:38.800 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509539 2024-08-19T15:03:38.800 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:38.800 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:03:38.800 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509539 2024-08-19T15:03:38.800 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:38.800 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509539 2024-08-19T15:03:38.800 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509539' 2024-08-19T15:03:38.800 INFO:tasks.workunit.client.0.smithi102.stderr:267201: waiting osd.2 seq 64424509539 2024-08-19T15:03:38.800 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:03:38.800 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509539 -lt 64424509539 2024-08-19T15:03:38.800 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:38.801 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-498216206339 2024-08-19T15:03:38.801 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:38.801 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:03:38.801 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-498216206339 2024-08-19T15:03:38.801 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:38.801 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206339 2024-08-19T15:03:38.801 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 498216206339' 2024-08-19T15:03:38.801 INFO:tasks.workunit.client.0.smithi102.stderr:267201: waiting osd.3 seq 498216206339 2024-08-19T15:03:38.801 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:03:38.801 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206339 -lt 498216206339 2024-08-19T15:03:38.801 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:38.801 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182481 2024-08-19T15:03:38.802 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:38.802 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-08-19T15:03:38.802 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182481 2024-08-19T15:03:38.802 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:38.802 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182481 2024-08-19T15:03:38.802 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182481' 2024-08-19T15:03:38.802 INFO:tasks.workunit.client.0.smithi102.stderr:267201: waiting osd.4 seq 107374182481 2024-08-19T15:03:38.802 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-08-19T15:03:38.802 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182481 -lt 107374182481 2024-08-19T15:03:38.802 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:38.802 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-511101108227 2024-08-19T15:03:38.967 INFO:tasks.workunit.client.0.smithi102.stderr:267201:01108226 2024-08-19T15:03:38.967 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 511101108226' 2024-08-19T15:03:38.967 INFO:tasks.workunit.client.0.smithi102.stderr:267199: waiting osd.5 seq 511101108226 2024-08-19T15:03:38.967 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T15:03:38.967 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 511101108227 -lt 511101108226 2024-08-19T15:03:38.967 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:38.967 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855430 2024-08-19T15:03:38.967 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:38.967 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T15:03:38.967 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855430 2024-08-19T15:03:38.967 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:38.968 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855430 2024-08-19T15:03:38.968 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855430' 2024-08-19T15:03:38.968 INFO:tasks.workunit.client.0.smithi102.stderr:267199: waiting osd.6 seq 150323855430 2024-08-19T15:03:38.968 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T15:03:38.968 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855432 -lt 150323855430 2024-08-19T15:03:38.968 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:38.968 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691900 2024-08-19T15:03:38.968 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:38.968 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T15:03:38.968 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691900 2024-08-19T15:03:38.968 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:38.969 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691900 2024-08-19T15:03:38.969 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691900' 2024-08-19T15:03:38.969 INFO:tasks.workunit.client.0.smithi102.stderr:267199: waiting osd.7 seq 171798691900 2024-08-19T15:03:38.969 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T15:03:38.969 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691901 -lt 171798691900 2024-08-19T15:03:38.969 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:38.969 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528375 2024-08-19T15:03:38.969 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:38.969 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T15:03:38.969 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528375 2024-08-19T15:03:38.969 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:38.969 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528375 2024-08-19T15:03:38.969 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528375' 2024-08-19T15:03:38.969 INFO:tasks.workunit.client.0.smithi102.stderr:267199: waiting osd.8 seq 193273528375 2024-08-19T15:03:38.970 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T15:03:39.727 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:39.727 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-08-19T15:03:39.727 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-511101108227 2024-08-19T15:03:39.727 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:39.727 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=511101108227 2024-08-19T15:03:39.727 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 511101108227' 2024-08-19T15:03:39.727 INFO:tasks.workunit.client.0.smithi102.stderr:267201: waiting osd.5 seq 511101108227 2024-08-19T15:03:39.727 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-08-19T15:03:39.727 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 511101108227 -lt 511101108227 2024-08-19T15:03:39.727 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:39.727 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855431 2024-08-19T15:03:39.728 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:39.728 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-08-19T15:03:39.728 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855431 2024-08-19T15:03:39.728 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:39.728 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855431 2024-08-19T15:03:39.728 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855431' 2024-08-19T15:03:39.728 INFO:tasks.workunit.client.0.smithi102.stderr:267201: waiting osd.6 seq 150323855431 2024-08-19T15:03:39.728 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-08-19T15:03:39.728 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855432 -lt 150323855431 2024-08-19T15:03:39.728 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:39.728 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691901 2024-08-19T15:03:39.728 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:39.728 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-08-19T15:03:39.729 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691901 2024-08-19T15:03:39.729 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:39.729 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691901 2024-08-19T15:03:39.729 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691901' 2024-08-19T15:03:39.729 INFO:tasks.workunit.client.0.smithi102.stderr:267201: waiting osd.7 seq 171798691901 2024-08-19T15:03:39.729 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-08-19T15:03:39.729 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691901 -lt 171798691901 2024-08-19T15:03:39.729 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:39.729 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528376 2024-08-19T15:03:39.729 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:39.729 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-08-19T15:03:39.729 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528376 2024-08-19T15:03:40.254 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: f93273528376 -lt 193273528375 2024-08-19T15:03:40.254 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:40.254 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631501 2024-08-19T15:03:40.255 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:40.255 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T15:03:40.255 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631501 2024-08-19T15:03:40.255 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:40.255 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631501 2024-08-19T15:03:40.255 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631501' 2024-08-19T15:03:40.255 INFO:tasks.workunit.client.0.smithi102.stderr:267199: waiting osd.9 seq 442381631501 2024-08-19T15:03:40.255 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T15:03:40.255 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631502 -lt 442381631501 2024-08-19T15:03:40.255 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:03:40.255 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:03:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:03:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:03:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:03:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:03:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:03:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:03:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:03:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:03:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:03:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:03:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:03:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:03:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:267199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:03:40.257 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:03:40.257 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:03:40.257 INFO:tasks.workunit.client.0.smithi102.stderr:267199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:03:40.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T15:03:40.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T15:03:40.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 267198 2024-08-19T15:03:41.286 INFO:tasks.workunit.client.0.smithi102.stderr:lush_pg_stats: cut -d - -f 2 2024-08-19T15:03:41.286 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528376 2024-08-19T15:03:41.286 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528376' 2024-08-19T15:03:41.286 INFO:tasks.workunit.client.0.smithi102.stderr:267201: waiting osd.8 seq 193273528376 2024-08-19T15:03:41.287 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-08-19T15:03:41.287 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528377 -lt 193273528376 2024-08-19T15:03:41.287 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:03:41.287 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631502 2024-08-19T15:03:41.287 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:03:41.287 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-08-19T15:03:41.287 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631502 2024-08-19T15:03:41.287 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:03:41.287 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631502 2024-08-19T15:03:41.287 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631502' 2024-08-19T15:03:41.287 INFO:tasks.workunit.client.0.smithi102.stderr:267201: waiting osd.9 seq 442381631502 2024-08-19T15:03:41.287 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-08-19T15:03:41.287 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631502 -lt 442381631502 2024-08-19T15:03:41.288 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:03:41.288 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:03:41.288 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:03:41.288 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:03:41.288 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:03:41.288 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:03:41.288 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:03:41.288 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:03:41.288 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:03:41.288 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:03:41.288 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:03:41.289 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:03:41.289 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:03:41.289 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:03:41.289 INFO:tasks.workunit.client.0.smithi102.stderr:267201: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:03:41.289 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:03:41.289 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:03:41.289 INFO:tasks.workunit.client.0.smithi102.stderr:267201: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:03:41.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T15:03:41.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-08-19T15:03:41.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-08-19T15:03:41.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-08-19T15:03:41.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-08-19T15:03:41.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-08-19T15:03:41.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T15:03:41.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T15:03:41.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T15:03:41.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:03:41.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:03:41.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:03:41.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:03:41.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:03:41.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:03:41.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:03:41.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:03:41.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:03:41.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:03:41.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:03:41.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:03:41.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:03:41.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:03:41.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:03:41.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:03:41.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:03:41.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:03:41.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:03:41.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:03:41.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:03:41.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:03:41.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:03:41.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:03:41.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:03:41.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:03:41.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:03:41.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:03:41.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T15:03:41.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T15:03:41.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T15:03:41.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T15:03:41.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:03:41.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:03:41.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:03:41.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:03:41.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:03:41.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:03:41.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:03:41.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:03:41.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:03:41.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:03:41.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:03:41.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:03:41.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:03:41.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:03:41.534 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:03:41.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:03:41.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:03:41.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:03:41.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:03:41.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:03:41.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:03:41.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:03:41.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:03:41.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:03:41.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:03:41.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:03:41.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:03:41.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:03:41.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T15:03:41.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T15:03:41.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:03:41.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:03:41.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T15:03:41.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T15:03:41.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T15:03:41.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T15:03:41.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T15:03:41.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2024-08-19T15:03:41.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2024-08-19T15:03:41.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:95: TEST_corrupt_and_repair_replicated: local poolname=rbd 2024-08-19T15:03:41.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-08-19T15:03:41.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T15:03:41.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T15:03:41.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T15:03:41.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T15:03:41.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T15:03:41.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-08-19T15:03:41.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T15:03:41.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:03:41.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:03:41.908 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:03:41.908 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:03:41.909 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:03:41.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:03:41.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-08-19T15:03:42.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T15:03:42.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T15:03:42.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:03:42.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:03:42.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T15:03:42.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T15:03:42.110 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:03:42.110 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:03:42.111 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:03:42.111 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:03:42.111 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:03:42.111 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:03:42.111 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:03:42.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:03:42.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T15:03:42.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T15:03:42.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:03:42.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:03:42.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T15:03:42.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T15:03:42.189 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:03:42.189 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:03:42.189 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:03:42.189 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:03:42.189 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:03:42.189 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:03:42.189 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:03:42.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:03:42.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T15:03:42.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2024-08-19T15:03:42.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T15:03:42.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T15:03:42.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T15:03:42.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T15:03:42.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T15:03:42.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T15:03:43.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T15:03:43.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:03:43.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:03:43.076 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:03:43.077 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:03:43.077 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:03:43.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:03:43.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:03:43.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:03:43.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2024-08-19T15:03:43.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:03:43.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:03:43.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T15:03:43.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:03:43.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:03:43.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:03:43.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:03:43.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:03:43.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:03:43.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:03:43.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:03:43.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:03:43.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:03:43.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:03:43.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:03:43.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:03:43.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:03:43.109 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:03:43.109 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:03:43.109 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:03:43.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:03:43.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:03:43.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:03:43.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:03:43.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:03:43.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:03:43.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:03:43.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:03:43.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:03:43.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:03:43.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:03:43.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:03:43.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:03:43.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:03:43.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:03:43.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7ee78d86-fedf-4dd0-9d2b-044544de3b9d 2024-08-19T15:03:43.111 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 7ee78d86-fedf-4dd0-9d2b-044544de3b9d 2024-08-19T15:03:43.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7ee78d86-fedf-4dd0-9d2b-044544de3b9d' 2024-08-19T15:03:43.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:03:43.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDPXsNm/+OQBxAAvnuIJAoqOMhk8x5s0yj9VA== 2024-08-19T15:03:43.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDPXsNm/+OQBxAAvnuIJAoqOMhk8x5s0yj9VA=="}' 2024-08-19T15:03:43.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7ee78d86-fedf-4dd0-9d2b-044544de3b9d -i td/osd-scrub-repair/0/new.json 2024-08-19T15:03:43.316 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:03:43.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T15:03:43.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPXsNm/+OQBxAAvnuIJAoqOMhk8x5s0yj9VA== --osd-uuid 7ee78d86-fedf-4dd0-9d2b-044544de3b9d 2024-08-19T15:03:43.356 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:03:43.355+0000 7f4672a1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:43.367 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:03:43.367+0000 7f4672a1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:43.368 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:03:43.368+0000 7f4672a1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:43.369 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:03:43.368+0000 7f4672a1c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:03:43.369 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:03:43.368+0000 7f4672a1c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:03:43.369 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:03:43.368+0000 7f4672a1c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:03:43.451 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:03:43.450+0000 7f4672a1c640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T15:03:48.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T15:03:48.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:03:48.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T15:03:48.958 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T15:03:48.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:03:49.308 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:03:49.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T15:03:49.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:03:49.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:03:49.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:03:49.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:03:49.335 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:03:49.335+0000 7f3287add640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:49.337 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:03:49.337+0000 7f3287add640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:49.338 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:03:49.338+0000 7f3287add640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:03:49.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T15:03:49.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:03:49.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:03:49.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:03:49.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:03:49.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:49.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:03:49.574 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:03:49.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:49.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:03:49.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:49.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:03:49.861+0000 7f3287add640 -1 Falling back to public interface 2024-08-19T15:03:50.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:50.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:50.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:03:50.838 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:03:50.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:50.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:03:51.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:52.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:52.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:52.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:03:52.111 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:03:52.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:52.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:03:52.187 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:03:52.185+0000 7f3287add640 -1 osd.0 0 log_to_monitors true 2024-08-19T15:03:52.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:53.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:53.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:53.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:03:53.387 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:03:53.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:53.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:03:53.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:54.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:54.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:54.664 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:03:54.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:03:54.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:54.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:03:54.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:55.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:55.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:55.973 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:03:55.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:03:55.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:55.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:03:56.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:57.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:57.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:57.248 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:03:57.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:03:57.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:57.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:03:57.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:58.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:58.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:58.526 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:03:58.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:03:58.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:58.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:03:58.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:03:59.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:03:59.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:03:59.808 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:03:59.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:03:59.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:03:59.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:04:00.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:01.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:01.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:01.143 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:04:01.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:04:01.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:01.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:04:01.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:02.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:02.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:02.424 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:04:02.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:04:02.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:02.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:04:02.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:03.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:03.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:03.697 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:04:03.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:04:03.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:03.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:04:03.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:04.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:04.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:04.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:04:04.997 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:04:04.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:04.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:04:05.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:06.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:06.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:06.320 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:04:06.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:04:06.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:06.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:04:06.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:07.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:07.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:07.855 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:04:07.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:04:07.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:07.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:04:08.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:09.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:09.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:09.616 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:04:09.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:04:09.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:09.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:04:10.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:11.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:11.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:11.444 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:04:11.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:04:11.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:11.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:04:11.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:12.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:12.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:12.723 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:04:12.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:04:12.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:12.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:04:13.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:14.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:14.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:14.393 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:04:14.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:04:14.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:14.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:04:14.743 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2419984699,v1:127.0.0.1:6803/2419984699] [v2:127.0.0.1:6804/2419984699,v1:127.0.0.1:6805/2419984699] exists,up 7ee78d86-fedf-4dd0-9d2b-044544de3b9d 2024-08-19T15:04:14.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:04:14.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:04:14.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:04:14.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2024-08-19T15:04:14.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:04:14.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:04:14.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T15:04:14.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:04:14.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:04:14.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:04:14.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:04:14.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:04:14.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:04:14.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:04:14.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:04:14.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:04:14.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:04:14.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:04:14.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:04:14.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:04:14.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:04:14.749 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:04:14.749 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:04:14.749 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:04:14.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:04:14.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:04:14.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:04:14.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:04:14.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:04:14.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:04:14.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:04:14.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:04:14.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:04:14.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:04:14.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:04:14.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:04:14.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:04:14.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:04:14.750 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:04:14.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dfd9a137-d3d4-4f33-9bd1-137f44aafd0f 2024-08-19T15:04:14.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 dfd9a137-d3d4-4f33-9bd1-137f44aafd0f' 2024-08-19T15:04:14.751 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 dfd9a137-d3d4-4f33-9bd1-137f44aafd0f 2024-08-19T15:04:14.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:04:14.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDuXsNmoQ2VLRAAD1dyT3ZQAslqzbb0H5CwZA== 2024-08-19T15:04:14.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDuXsNmoQ2VLRAAD1dyT3ZQAslqzbb0H5CwZA=="}' 2024-08-19T15:04:14.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dfd9a137-d3d4-4f33-9bd1-137f44aafd0f -i td/osd-scrub-repair/1/new.json 2024-08-19T15:04:15.175 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:04:15.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T15:04:15.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDuXsNmoQ2VLRAAD1dyT3ZQAslqzbb0H5CwZA== --osd-uuid dfd9a137-d3d4-4f33-9bd1-137f44aafd0f 2024-08-19T15:04:15.216 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:04:15.215+0000 7f84a4015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:04:15.256 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:04:15.255+0000 7f84a4015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:04:15.257 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:04:15.256+0000 7f84a4015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:04:15.257 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:04:15.256+0000 7f84a4015640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:04:15.258 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:04:15.256+0000 7f84a4015640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:04:15.258 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:04:15.256+0000 7f84a4015640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:04:15.318 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:04:15.317+0000 7f84a4015640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T15:04:20.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T15:04:20.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:04:20.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T15:04:20.774 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T15:04:20.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:04:21.118 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:04:21.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T15:04:21.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:04:21.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:04:21.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:04:21.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:04:21.145 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:04:21.144+0000 7fa149414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:04:21.147 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:04:21.146+0000 7fa149414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:04:21.148 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:04:21.148+0000 7fa149414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:04:21.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T15:04:21.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:04:21.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:04:21.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:04:21.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:04:21.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:21.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:04:21.414 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:04:21.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:21.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:04:21.661 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:04:21.660+0000 7fa149414640 -1 Falling back to public interface 2024-08-19T15:04:21.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:22.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:22.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:22.682 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:04:22.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:04:22.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:22.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:04:22.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:23.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:23.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:23.965 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:04:23.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:04:23.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:23.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:04:24.016 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:04:24.015+0000 7fa149414640 -1 osd.1 0 log_to_monitors true 2024-08-19T15:04:24.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:25.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:25.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:25.236 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:04:25.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:04:25.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:25.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:04:25.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:26.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:26.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:26.634 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:04:26.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:04:26.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:26.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:04:26.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:27.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:27.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:27.972 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:04:27.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:04:27.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:27.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:04:28.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:29.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:29.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:29.256 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:04:29.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:04:29.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:29.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:04:29.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:30.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:30.620 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:04:30.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:30.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:04:30.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:30.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:04:30.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:31.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:31.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:31.896 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:04:31.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:04:31.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:31.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:04:32.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:33.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:33.177 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:04:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:04:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:04:33.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:34.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:34.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:34.452 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:04:34.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:04:34.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:34.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:04:34.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:35.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:35.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:35.809 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:04:35.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:04:35.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:35.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:04:36.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:37.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:37.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:37.320 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:04:37.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:04:37.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:37.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:04:37.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:38.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:38.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:38.889 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:04:38.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:04:38.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:38.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:04:39.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:40.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:40.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:40.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:04:40.266 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:04:40.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:40.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:04:40.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:41.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:41.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:41.737 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:04:41.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:04:41.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:41.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:04:42.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:43.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:43.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:43.432 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:04:43.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:04:43.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:43.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:04:44.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:45.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:45.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:45.333 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:04:45.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:04:45.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:45.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:04:45.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:46.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:04:46.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:46.689 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:04:46.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:04:46.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:46.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:04:46.959 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1977463638,v1:127.0.0.1:6811/1977463638] [v2:127.0.0.1:6812/1977463638,v1:127.0.0.1:6813/1977463638] exists,up dfd9a137-d3d4-4f33-9bd1-137f44aafd0f 2024-08-19T15:04:46.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:04:46.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:04:46.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:04:46.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: create_rbd_pool 2024-08-19T15:04:46.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T15:04:47.339 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-08-19T15:04:47.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T15:04:47.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T15:04:47.697 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' created 2024-08-19T15:04:47.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:04:48.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T15:04:50.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:102: TEST_corrupt_and_repair_replicated: wait_for_clean 2024-08-19T15:04:50.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:04:50.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:04:50.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:04:50.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:04:50.401 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:04:50.401 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:04:50.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:04:50.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:04:50.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:04:50.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:04:50.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:04:50.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:04:50.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:04:50.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:04:50.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:04:50.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:04:50.781 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:04:50.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:04:50.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:04:50.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:04:50.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836492 2024-08-19T15:04:50.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836492 2024-08-19T15:04:50.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492' 2024-08-19T15:04:50.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:04:50.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:04:51.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672968 2024-08-19T15:04:51.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672968 2024-08-19T15:04:51.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492 1-42949672968' 2024-08-19T15:04:51.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:04:51.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836492 2024-08-19T15:04:51.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:04:51.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:04:51.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836492 2024-08-19T15:04:51.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:04:51.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836492 2024-08-19T15:04:51.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-08-19T15:04:51.031 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836492 2024-08-19T15:04:51.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:04:51.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836492 2024-08-19T15:04:51.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:04:51.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672968 2024-08-19T15:04:51.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:04:51.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:04:51.301 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672968 2024-08-19T15:04:51.301 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:04:51.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672968 2024-08-19T15:04:51.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-08-19T15:04:51.302 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672968 2024-08-19T15:04:51.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:04:51.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672968 2024-08-19T15:04:51.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:04:52.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:04:52.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:04:52.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672968 2024-08-19T15:04:52.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:04:53.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:04:53.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:04:54.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672969 -lt 42949672968 2024-08-19T15:04:54.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:04:54.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:04:54.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:04:54.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T15:04:54.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:04:54.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:04:54.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:04:54.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:04:54.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:04:54.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:04:54.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:04:54.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:04:54.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:04:54.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:04:54.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:04:55.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T15:04:55.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:04:55.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:04:55.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2024-08-19T15:04:55.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:04:55.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-08-19T15:04:55.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-08-19T15:04:55.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:04:55.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:04:55.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:04:55.545 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:04:55.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:04:56.031 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:04:56.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:04:56.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:04:56.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-08-19T15:04:56.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2024-08-19T15:04:56.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-08-19T15:04:56.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-08-19T15:04:56.271 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-08-19T15:04:56.271 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-08-19T15:04:56.271 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T15:04:56.271 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-08-19T15:04:56.271 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:04:56.536 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T15:04:56.536 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-08-19T15:04:56.536 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T15:04:56.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-08-19T15:04:56.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-08-19T15:04:56.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-08-19T15:04:56.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-08-19T15:04:56.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T15:04:56.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:04:56.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:04:56.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:04:56.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:04:56.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T15:04:56.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:04:56.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:04:56.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:04:56.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:04:56.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:04:56.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:04:56.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:04:56.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:04:56.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:04:56.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:04:57.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:04:57.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T15:04:57.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:04:57.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:04:57.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:04:57.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:04:57.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:04:57.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-08-19T15:04:58.588 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-08-19T15:04:59.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:04:59.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:04:59.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:04:59.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:04:59.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:04:59.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:04:59.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:04:59.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:04:59.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:04:59.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:04:59.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:04:59.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:04:59.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:04:59.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:04:59.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:04:59.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:04:59.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:04:59.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:04:59.110 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:04:59.110 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:04:59.110 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:04:59.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:04:59.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:04:59.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:04:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:04:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:04:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:04:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:04:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:04:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:04:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:04:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:04:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:04:59.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:04:59.112 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:04:59.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:04:59.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:04:59.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:04:59.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:04:59.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:04:59.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:04:59.138 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:04:59.138+0000 7f7c05d2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:04:59.140 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:04:59.140+0000 7f7c05d2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:04:59.142 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:04:59.141+0000 7f7c05d2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:04:59.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:04:59.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:04:59.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:04:59.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:04:59.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:04:59.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:04:59.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:04:59.397 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:04:59.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:04:59.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:04:59.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:04:59.675 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:04:59.675+0000 7f7c05d2c640 -1 Falling back to public interface 2024-08-19T15:05:00.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:05:00.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:05:00.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:05:00.664 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:05:00.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:05:00.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:05:00.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:05:01.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:05:01.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:05:01.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:05:01.938 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:05:01.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:05:01.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:05:02.051 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:02.050+0000 7f7c05d2c640 -1 osd.0 21 log_to_monitors true 2024-08-19T15:05:02.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:05:03.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:05:03.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:05:03.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:05:03.266 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:05:03.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:05:03.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:05:03.608 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3908667099,v1:127.0.0.1:6803/3908667099] [v2:127.0.0.1:6804/3908667099,v1:127.0.0.1:6805/3908667099] exists,up 7ee78d86-fedf-4dd0-9d2b-044544de3b9d 2024-08-19T15:05:03.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:05:03.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:05:03.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:05:03.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:05:03.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:05:03.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:05:03.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:05:03.609 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:05:03.609 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:05:03.609 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:05:03.609 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:05:03.609 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:05:03.609 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:05:03.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:05:03.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:05:03.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:05:03.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:05:03.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:05:03.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:05:03.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:05:03.961 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:05:03.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:05:03.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:05:03.962 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:05:04.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149698 2024-08-19T15:05:04.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149698 2024-08-19T15:05:04.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698' 2024-08-19T15:05:04.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:05:04.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:05:04.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-08-19T15:05:04.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-08-19T15:05:04.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698 1-42949672972' 2024-08-19T15:05:04.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:05:04.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149698 2024-08-19T15:05:04.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:05:04.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:05:04.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149698 2024-08-19T15:05:04.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:05:04.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149698 2024-08-19T15:05:04.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-08-19T15:05:04.212 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 111669149698 2024-08-19T15:05:04.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:05:04.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149698 2024-08-19T15:05:04.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:05:05.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:05:05.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:05:05.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149698 -lt 111669149698 2024-08-19T15:05:05.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:05:05.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-08-19T15:05:05.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:05:05.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:05:05.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-08-19T15:05:05.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:05:05.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-08-19T15:05:05.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-08-19T15:05:05.758 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949672972 2024-08-19T15:05:05.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:05:06.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-08-19T15:05:06.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:05:06.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:05:06.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:05:06.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T15:05:06.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:05:06.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:05:06.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:05:06.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:05:06.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:05:06.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:05:06.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:05:06.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:05:06.622 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:05:06.622 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:05:06.622 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:05:06.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T15:05:06.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:05:06.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:05:06.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-08-19T15:05:06.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-08-19T15:05:06.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T15:05:06.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-08-19T15:05:06.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:05:07.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-08-19T15:05:07.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-08-19T15:05:07.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-08-19T15:05:07.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-08-19T15:05:07.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:05:07.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:05:07.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:05:07.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:05:07.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T15:04:47.660320+0000 2024-08-19T15:05:07.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-08-19T15:05:07.761 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 1.3 on osd.1 to repair 2024-08-19T15:05:07.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-08-19T15:04:47.660320+0000 2024-08-19T15:05:07.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-08-19T15:05:07.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:04:47.660320+0000 2024-08-19T15:05:07.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:05:07.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:05:07.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:05:07.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T15:05:07.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:05:07.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:05:07.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:05:07.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:05:08.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:04:47.660320+0000 '>' 2024-08-19T15:04:47.660320+0000 2024-08-19T15:05:08.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:05:09.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:05:09.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:05:09.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T15:05:09.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:05:09.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:05:09.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:05:09.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:05:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:04:47.660320+0000 '>' 2024-08-19T15:04:47.660320+0000 2024-08-19T15:05:09.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:05:10.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:05:10.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:05:10.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T15:05:10.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:05:10.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:05:10.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:05:10.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:05:10.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:04:47.660320+0000 '>' 2024-08-19T15:04:47.660320+0000 2024-08-19T15:05:10.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:05:11.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:05:11.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:05:11.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T15:05:11.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:05:11.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:05:11.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:05:11.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:05:11.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:04:47.660320+0000 '>' 2024-08-19T15:04:47.660320+0000 2024-08-19T15:05:11.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:05:12.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:05:12.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:05:12.862 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T15:05:12.862 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:05:12.862 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:05:12.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:05:12.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:05:13.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:05:08.006234+0000 '>' 2024-08-19T15:04:47.660320+0000 2024-08-19T15:05:13.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:05:13.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T15:05:13.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:05:13.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:05:13.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:05:13.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:05:13.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T15:05:13.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:05:13.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:05:13.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:05:13.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:05:13.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:05:13.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:05:13.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:05:13.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:05:13.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:05:13.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:05:13.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:05:13.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T15:05:13.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:05:13.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:05:13.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:05:13.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:05:13.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:05:13.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-08-19T15:05:14.668 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-08-19T15:05:14.668 INFO:tasks.workunit.client.0.smithi102.stdout:_ 2024-08-19T15:05:14.668 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 2024-08-19T15:05:14.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:05:14.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:05:14.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:05:14.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:05:14.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:05:14.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:05:14.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:05:14.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:05:14.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:05:14.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:05:14.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:05:14.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:05:14.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:05:14.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:05:14.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:05:14.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:05:14.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:05:14.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:05:14.944 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:05:14.944 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:05:14.945 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:05:14.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:05:14.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:05:14.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:05:14.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:05:14.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:05:14.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:05:14.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:05:14.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:05:14.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:05:14.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:05:14.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:05:14.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:05:14.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:05:14.946 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:05:14.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:05:14.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:05:14.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:05:14.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:05:14.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:05:14.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:05:14.973 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:14.972+0000 7fb94fc1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:05:14.976 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:14.975+0000 7fb94fc1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:05:14.977 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:14.976+0000 7fb94fc1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:05:15.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:05:15.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:05:15.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:05:15.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:05:15.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:05:15.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:05:15.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:05:15.243 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:05:15.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:05:15.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:05:15.491 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:15.491+0000 7fb94fc1c640 -1 Falling back to public interface 2024-08-19T15:05:15.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:05:16.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:05:16.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:05:16.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:05:16.512 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:05:16.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:05:16.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:05:16.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:05:17.653 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:17.652+0000 7fb94fc1c640 -1 osd.1 28 log_to_monitors true 2024-08-19T15:05:17.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:05:17.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:05:17.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:05:17.786 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:05:17.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:05:17.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:05:18.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:05:19.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:05:19.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:05:19.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:05:19.058 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:05:19.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:05:19.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:05:19.392 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/2014204295,v1:127.0.0.1:6811/2014204295] [v2:127.0.0.1:6812/2014204295,v1:127.0.0.1:6813/2014204295] exists,up dfd9a137-d3d4-4f33-9bd1-137f44aafd0f 2024-08-19T15:05:19.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:05:19.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:05:19.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:05:19.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:05:19.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:05:19.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:05:19.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:05:19.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:05:19.393 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:05:19.393 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:05:19.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:05:19.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:05:19.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:05:19.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:05:19.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:05:19.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:05:19.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:05:19.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:05:19.484 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:05:19.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:05:19.748 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:05:19.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:05:19.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:05:19.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:05:19.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149702 2024-08-19T15:05:19.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149702 2024-08-19T15:05:19.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702' 2024-08-19T15:05:19.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:05:19.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:05:19.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-08-19T15:05:19.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-08-19T15:05:19.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-08-19T15:05:19.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:05:19.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149702 2024-08-19T15:05:19.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:05:19.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:05:19.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149702 2024-08-19T15:05:19.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:05:19.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149702 2024-08-19T15:05:19.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-08-19T15:05:19.957 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 111669149702 2024-08-19T15:05:19.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:05:20.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149701 -lt 111669149702 2024-08-19T15:05:20.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:05:21.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:05:21.224 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:05:21.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149702 -lt 111669149702 2024-08-19T15:05:21.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:05:21.497 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-137438953474 2024-08-19T15:05:21.497 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:05:21.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:05:21.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-137438953474 2024-08-19T15:05:21.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:05:21.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-08-19T15:05:21.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-08-19T15:05:21.500 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 137438953474 2024-08-19T15:05:21.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:05:21.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953474 -lt 137438953474 2024-08-19T15:05:21.767 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:05:21.767 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:05:21.767 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:05:22.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T15:05:22.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:05:22.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:05:22.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:05:22.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:05:22.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:05:22.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:05:22.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:05:22.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:05:22.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:05:22.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:05:22.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:05:22.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T15:05:22.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:05:22.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:05:22.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T15:05:22.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T15:05:22.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2024-08-19T15:05:22.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-08-19T15:05:22.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T15:05:22.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-08-19T15:05:22.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:05:23.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2024-08-19T15:05:23.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-08-19T15:05:23.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-08-19T15:05:23.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-08-19T15:05:23.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T15:05:23.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:05:23.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:05:23.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:05:23.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:05:23.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T15:05:23.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:05:23.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:05:23.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:05:23.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:05:23.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:05:23.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:05:23.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:05:23.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:05:23.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:05:23.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:05:23.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:05:23.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-08-19T15:05:23.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:05:23.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:05:23.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:05:23.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:05:23.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:05:23.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-08-19T15:05:24.375 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-08-19T15:05:24.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:05:24.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:05:24.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:05:24.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:05:24.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:05:24.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:05:24.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:05:24.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:05:24.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:05:24.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:05:24.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:05:24.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:05:24.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:05:24.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:05:24.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:05:24.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:05:24.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:05:24.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:05:24.893 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:05:24.893 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:05:24.893 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:05:24.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:05:24.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:05:24.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:05:24.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:05:24.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:05:24.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:05:24.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:05:24.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:05:24.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:05:24.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:05:24.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:05:24.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:05:24.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:05:24.895 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:05:24.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:05:24.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:05:24.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:05:24.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:05:24.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:05:24.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:05:24.922 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:24.921+0000 7f53eec1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:05:24.925 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:24.924+0000 7f53eec1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:05:24.926 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:24.925+0000 7f53eec1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:05:25.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:05:25.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:05:25.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:05:25.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:05:25.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:05:25.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:05:25.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:05:25.172 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:05:25.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:05:25.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:05:25.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:05:25.448 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:25.448+0000 7f53eec1e640 -1 Falling back to public interface 2024-08-19T15:05:26.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:05:26.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:05:26.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:05:26.438 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:05:26.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:05:26.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:05:26.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:05:27.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:05:27.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:05:27.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:05:27.713 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:05:27.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:05:27.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:05:27.752 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:27.751+0000 7f53eec1e640 -1 osd.1 34 log_to_monitors true 2024-08-19T15:05:27.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:05:28.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:05:28.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:05:28.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:05:28.989 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:05:28.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:05:28.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:05:29.294 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [32,34) [v2:127.0.0.1:6810/3909146915,v1:127.0.0.1:6811/3909146915] [v2:127.0.0.1:6812/3909146915,v1:127.0.0.1:6813/3909146915] exists,up dfd9a137-d3d4-4f33-9bd1-137f44aafd0f 2024-08-19T15:05:29.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:05:29.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:05:29.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:05:29.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:05:29.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:05:29.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:05:29.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:05:29.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:05:29.295 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:05:29.295 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:05:29.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:05:29.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:05:29.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:05:29.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:05:29.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:05:29.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:05:29.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:05:29.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:05:29.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:05:29.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:05:29.648 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:05:29.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:05:29.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:05:29.648 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:05:29.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149705 2024-08-19T15:05:29.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149705 2024-08-19T15:05:29.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149705' 2024-08-19T15:05:29.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:05:29.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:05:29.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757250 2024-08-19T15:05:29.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757250 2024-08-19T15:05:29.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149705 1-163208757250' 2024-08-19T15:05:29.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:05:29.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149705 2024-08-19T15:05:29.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:05:29.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:05:29.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149705 2024-08-19T15:05:29.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:05:29.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149705 2024-08-19T15:05:29.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149705' 2024-08-19T15:05:29.858 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 111669149705 2024-08-19T15:05:29.858 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:05:30.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149704 -lt 111669149705 2024-08-19T15:05:30.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:05:31.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:05:31.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:05:31.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149705 -lt 111669149705 2024-08-19T15:05:31.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:05:31.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-163208757250 2024-08-19T15:05:31.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:05:31.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:05:31.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-163208757250 2024-08-19T15:05:31.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:05:31.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757250 2024-08-19T15:05:31.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 163208757250' 2024-08-19T15:05:31.425 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 163208757250 2024-08-19T15:05:31.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:05:31.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757250 -lt 163208757250 2024-08-19T15:05:31.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:05:31.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:05:31.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:05:32.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T15:05:32.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:05:32.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:05:32.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:05:32.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:05:32.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:05:32.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:05:32.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:05:32.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:05:32.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:05:32.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:05:32.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:05:32.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T15:05:32.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:05:32.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:05:32.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-08-19T15:05:32.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-08-19T15:05:32.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T15:05:32.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-08-19T15:05:32.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:05:32.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-08-19T15:05:32.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-08-19T15:05:32.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-08-19T15:05:32.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-08-19T15:05:32.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:05:32.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:05:32.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:05:32.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:05:33.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T15:05:08.006234+0000 2024-08-19T15:05:33.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-08-19T15:05:33.434 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 1.3 on osd.1 to repair 2024-08-19T15:05:33.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-08-19T15:05:08.006234+0000 2024-08-19T15:05:33.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-08-19T15:05:33.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:05:08.006234+0000 2024-08-19T15:05:33.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:05:33.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:05:33.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:05:33.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T15:05:33.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:05:33.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:05:33.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:05:33.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:05:33.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:05:08.006234+0000 '>' 2024-08-19T15:05:08.006234+0000 2024-08-19T15:05:33.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:05:34.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:05:34.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:05:34.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T15:05:34.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:05:34.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:05:34.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:05:34.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:05:34.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:05:08.006234+0000 '>' 2024-08-19T15:05:08.006234+0000 2024-08-19T15:05:34.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:05:35.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:05:35.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:05:35.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T15:05:35.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:05:35.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:05:35.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:05:35.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:05:36.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:05:08.006234+0000 '>' 2024-08-19T15:05:08.006234+0000 2024-08-19T15:05:36.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:05:37.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:05:37.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:05:37.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T15:05:37.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:05:37.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:05:37.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:05:37.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:05:37.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:05:08.006234+0000 '>' 2024-08-19T15:05:08.006234+0000 2024-08-19T15:05:37.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:05:38.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:05:38.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:05:38.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T15:05:38.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:05:38.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:05:38.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:05:38.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:05:38.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:05:08.006234+0000 '>' 2024-08-19T15:05:08.006234+0000 2024-08-19T15:05:38.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:05:39.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:05:39.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:05:39.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T15:05:39.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:05:39.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:05:39.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:05:39.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:05:40.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:05:33.842464+0000 '>' 2024-08-19T15:05:08.006234+0000 2024-08-19T15:05:40.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:05:40.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T15:05:40.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:05:40.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:05:40.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:05:40.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:05:40.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T15:05:40.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:05:40.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:05:40.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:05:40.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:05:40.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:05:40.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:05:40.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:05:40.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:05:40.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:05:40.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:05:40.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:05:40.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-08-19T15:05:40.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:05:40.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:05:40.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:05:40.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:05:40.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:05:40.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-08-19T15:05:41.614 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-08-19T15:05:41.615 INFO:tasks.workunit.client.0.smithi102.stdout:_ 2024-08-19T15:05:41.615 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 2024-08-19T15:05:41.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:05:41.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:05:41.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:05:41.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:05:41.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:05:41.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:05:41.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:05:41.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:05:41.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:05:41.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:05:41.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:05:41.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:05:41.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:05:41.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:05:41.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:05:41.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:05:41.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:05:41.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:05:41.884 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:05:41.884 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:05:41.884 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:05:41.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:05:41.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:05:41.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:05:41.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:05:41.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:05:41.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:05:41.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:05:41.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:05:41.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:05:41.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:05:41.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:05:41.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:05:41.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:05:41.885 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:05:41.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:05:41.886 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:05:41.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:05:41.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:05:41.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:05:41.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:05:41.912 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:41.911+0000 7fe46a211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:05:41.914 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:41.914+0000 7fe46a211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:05:41.915 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:41.915+0000 7fe46a211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:05:42.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:05:42.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:05:42.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:05:42.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:05:42.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:05:42.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:05:42.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:05:42.160 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:05:42.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:05:42.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:05:42.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:05:42.441 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:42.441+0000 7fe46a211640 -1 Falling back to public interface 2024-08-19T15:05:43.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:05:43.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:05:43.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:05:43.425 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:05:43.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:05:43.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:05:43.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:05:44.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:05:44.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:05:44.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:05:44.694 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:05:44.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:05:44.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:05:44.718 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:44.717+0000 7fe46a211640 -1 osd.1 39 log_to_monitors true 2024-08-19T15:05:44.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:05:45.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:05:45.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:05:45.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:05:45.994 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:05:45.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:05:45.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:05:46.315 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 40 last_clean_interval [38,39) [v2:127.0.0.1:6810/3737129459,v1:127.0.0.1:6811/3737129459] [v2:127.0.0.1:6812/3737129459,v1:127.0.0.1:6813/3737129459] exists,up dfd9a137-d3d4-4f33-9bd1-137f44aafd0f 2024-08-19T15:05:46.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:05:46.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:05:46.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:05:46.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:05:46.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:05:46.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:05:46.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:05:46.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:05:46.317 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:05:46.317 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:05:46.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:05:46.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:05:46.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:05:46.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:05:46.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:05:46.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:05:46.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:05:46.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:05:46.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:05:46.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:05:46.670 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:05:46.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:05:46.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:05:46.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:05:46.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149709 2024-08-19T15:05:46.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149709 2024-08-19T15:05:46.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149709' 2024-08-19T15:05:46.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:05:46.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:05:46.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593730 2024-08-19T15:05:46.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593730 2024-08-19T15:05:46.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149709 1-184683593730' 2024-08-19T15:05:46.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:05:46.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149709 2024-08-19T15:05:46.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:05:46.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:05:46.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149709 2024-08-19T15:05:46.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:05:46.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149709 2024-08-19T15:05:46.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149709' 2024-08-19T15:05:46.909 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 111669149709 2024-08-19T15:05:46.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:05:47.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149709 -lt 111669149709 2024-08-19T15:05:47.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:05:47.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-184683593730 2024-08-19T15:05:47.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:05:47.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:05:47.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-184683593730 2024-08-19T15:05:47.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:05:47.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593730 2024-08-19T15:05:47.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 184683593730' 2024-08-19T15:05:47.179 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 184683593730 2024-08-19T15:05:47.179 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:05:47.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593730 -lt 184683593730 2024-08-19T15:05:47.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:05:47.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:05:47.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:05:47.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T15:05:47.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:05:47.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:05:47.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:05:47.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:05:47.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:05:47.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:05:47.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:05:48.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:05:48.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:05:48.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:05:48.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:05:48.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T15:05:48.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:05:48.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:05:48.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T15:05:48.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T15:05:48.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T15:05:48.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:05:48.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:05:48.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:05:48.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:05:48.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:05:48.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:05:48.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:05:48.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:05:48.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:05:48.522 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:05:48.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:05:48.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:05:48.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:05:48.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:05:48.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:05:48.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:05:48.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:05:48.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:05:48.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:05:48.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:05:48.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:05:48.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:05:48.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:05:48.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:05:48.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:05:48.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:05:48.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:05:48.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:05:48.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T15:05:48.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T15:05:48.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T15:05:48.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T15:05:48.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:05:48.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:05:48.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:05:48.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:05:48.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:05:48.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:05:48.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:05:48.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:05:48.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:05:48.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:05:48.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:05:48.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:05:48.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:05:48.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:05:48.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:05:48.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:05:48.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:05:48.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:05:48.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:05:48.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:05:48.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:05:48.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:05:48.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:05:48.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:05:48.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:05:48.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:05:48.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:05:48.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:05:48.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T15:05:48.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T15:05:48.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:05:48.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:05:48.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T15:05:48.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T15:05:48.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T15:05:48.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T15:05:48.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T15:05:48.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2024-08-19T15:05:48.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5746: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2024-08-19T15:05:48.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-08-19T15:05:48.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=false 2024-08-19T15:05:48.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-08-19T15:05:48.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-08-19T15:05:48.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-08-19T15:05:48.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T15:05:48.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T15:05:48.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T15:05:48.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T15:05:48.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T15:05:48.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-08-19T15:05:48.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T15:05:48.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:05:48.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:05:48.937 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:05:48.937 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:05:48.938 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:05:48.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:05:48.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T15:05:49.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T15:05:49.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T15:05:49.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:05:49.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:05:49.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T15:05:49.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T15:05:49.114 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:05:49.114 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:05:49.114 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:05:49.115 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:05:49.115 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:05:49.115 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:05:49.115 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:05:49.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:05:49.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T15:05:49.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T15:05:49.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:05:49.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:05:49.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T15:05:49.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T15:05:49.194 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:05:49.194 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:05:49.194 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:05:49.195 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:05:49.195 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:05:49.195 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:05:49.195 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:05:49.195 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:05:49.196 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T15:05:49.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-08-19T15:05:49.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T15:05:49.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T15:05:49.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T15:05:49.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T15:05:49.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T15:05:49.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T15:05:50.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T15:05:50.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:05:50.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:05:50.081 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:05:50.081 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:05:50.081 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:05:50.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:05:50.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:05:50.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:05:50.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-08-19T15:05:50.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-08-19T15:05:50.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-08-19T15:05:50.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:05:50.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:05:50.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T15:05:50.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:05:50.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:05:50.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:05:50.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:05:50.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:05:50.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:05:50.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:05:50.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:05:50.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:05:50.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:05:50.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:05:50.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:05:50.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:05:50.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:05:50.116 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:05:50.116 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:05:50.116 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:05:50.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:05:50.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:05:50.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:05:50.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:05:50.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:05:50.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:05:50.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:05:50.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:05:50.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:05:50.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:05:50.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:05:50.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:05:50.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:05:50.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:05:50.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:05:50.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=835f901e-853d-4440-b3ef-eda73796c506 2024-08-19T15:05:50.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 835f901e-853d-4440-b3ef-eda73796c506' 2024-08-19T15:05:50.118 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 835f901e-853d-4440-b3ef-eda73796c506 2024-08-19T15:05:50.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:05:50.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBOX8Nmhk3fBxAAaVVCqyJWuXby44fw94v1hw== 2024-08-19T15:05:50.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBOX8Nmhk3fBxAAaVVCqyJWuXby44fw94v1hw=="}' 2024-08-19T15:05:50.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 835f901e-853d-4440-b3ef-eda73796c506 -i td/osd-scrub-repair/0/new.json 2024-08-19T15:05:50.338 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:05:50.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T15:05:50.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBOX8Nmhk3fBxAAaVVCqyJWuXby44fw94v1hw== --osd-uuid 835f901e-853d-4440-b3ef-eda73796c506 2024-08-19T15:05:50.378 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:50.376+0000 7feb70a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:05:50.380 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:50.379+0000 7feb70a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:05:50.381 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:50.381+0000 7feb70a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:05:50.382 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:50.381+0000 7feb70a18640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:05:50.382 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:50.381+0000 7feb70a18640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:05:50.382 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:50.381+0000 7feb70a18640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:05:50.463 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:50.463+0000 7feb70a18640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T15:05:55.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T15:05:55.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:05:55.869 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T15:05:55.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T15:05:55.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:05:56.216 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:05:56.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T15:05:56.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:05:56.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:05:56.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:05:56.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:05:56.243 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:56.242+0000 7f0b41f57640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:05:56.254 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:56.253+0000 7f0b41f57640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:05:56.255 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:56.254+0000 7f0b41f57640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:05:56.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T15:05:56.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:05:56.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:05:56.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:05:56.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:05:56.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:05:56.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:05:56.484 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:05:56.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:05:56.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:05:56.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:05:56.776 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:56.776+0000 7f0b41f57640 -1 Falling back to public interface 2024-08-19T15:05:57.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:05:57.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:05:57.749 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:05:57.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:05:57.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:05:57.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:05:58.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:05:59.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:05:59.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:05:59.025 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:05:59.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:05:59.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:05:59.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:05:59.038 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:05:59.037+0000 7f0b41f57640 -1 osd.0 0 log_to_monitors true 2024-08-19T15:05:59.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:00.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:00.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:00.439 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:06:00.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:06:00.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:00.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:06:00.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:01.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:01.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:01.732 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:06:01.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:06:01.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:01.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:06:02.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:03.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:03.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:03.008 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:06:03.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:06:03.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:03.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:06:03.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:04.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:04.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:04.284 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:06:04.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:06:04.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:04.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:06:04.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:05.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:05.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:05.562 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:06:05.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:06:05.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:05.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:06:05.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:06.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:06.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:06.914 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:06:06.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:06:06.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:06.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:06:07.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:08.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:08.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:08.188 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:06:08.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:06:08.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:08.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:06:08.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:09.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:09.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:09.463 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:06:09.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:06:09.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:09.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:06:09.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:10.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:10.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:10.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:06:10.890 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:06:10.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:10.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:06:11.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:12.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:12.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:12.600 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:06:12.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:06:12.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:12.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:06:13.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:14.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:14.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:14.153 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:06:14.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:06:14.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:14.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:06:15.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:16.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:16.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:16.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:06:16.041 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:06:16.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:16.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:06:16.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:17.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:17.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:17.916 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:06:17.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:06:17.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:17.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:06:18.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:19.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:19.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:19.283 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:06:19.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:06:19.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:19.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:06:19.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:20.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:20.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:20.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:06:20.784 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:06:20.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:20.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:06:21.057 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4269387301,v1:127.0.0.1:6803/4269387301] [v2:127.0.0.1:6804/4269387301,v1:127.0.0.1:6805/4269387301] exists,up 835f901e-853d-4440-b3ef-eda73796c506 2024-08-19T15:06:21.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:06:21.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:06:21.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:06:21.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-08-19T15:06:21.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-08-19T15:06:21.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:06:21.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:06:21.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T15:06:21.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:06:21.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:06:21.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:06:21.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:06:21.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:06:21.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:06:21.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:06:21.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:06:21.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:06:21.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:06:21.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:06:21.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:06:21.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:06:21.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:06:21.063 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:06:21.063 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:06:21.063 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:06:21.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:06:21.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:06:21.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:06:21.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:06:21.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:06:21.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:06:21.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:06:21.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:06:21.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:06:21.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:06:21.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:06:21.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:06:21.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:06:21.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:06:21.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:06:21.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d4a09a65-a9c7-4c80-8b37-aa2cb0bd38f9 2024-08-19T15:06:21.065 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 d4a09a65-a9c7-4c80-8b37-aa2cb0bd38f9 2024-08-19T15:06:21.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d4a09a65-a9c7-4c80-8b37-aa2cb0bd38f9' 2024-08-19T15:06:21.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:06:21.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBtX8NmHM+xBBAA+lP7qfJWxB3ZKso5ToVLCQ== 2024-08-19T15:06:21.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBtX8NmHM+xBBAA+lP7qfJWxB3ZKso5ToVLCQ=="}' 2024-08-19T15:06:21.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d4a09a65-a9c7-4c80-8b37-aa2cb0bd38f9 -i td/osd-scrub-repair/1/new.json 2024-08-19T15:06:21.387 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:06:21.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T15:06:21.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBtX8NmHM+xBBAA+lP7qfJWxB3ZKso5ToVLCQ== --osd-uuid d4a09a65-a9c7-4c80-8b37-aa2cb0bd38f9 2024-08-19T15:06:21.428 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:21.427+0000 7f2163014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:06:21.480 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:21.479+0000 7f2163014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:06:21.481 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:21.480+0000 7f2163014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:06:21.482 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:21.480+0000 7f2163014640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:06:21.482 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:21.480+0000 7f2163014640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:06:21.482 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:21.480+0000 7f2163014640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:06:21.542 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:21.541+0000 7f2163014640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T15:06:27.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T15:06:27.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:06:27.312 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T15:06:27.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T15:06:27.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:06:27.657 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:06:27.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T15:06:27.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:06:27.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:06:27.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:06:27.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:06:27.684 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:27.683+0000 7f6de3d99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:06:27.685 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:27.685+0000 7f6de3d99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:06:27.686 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:27.686+0000 7f6de3d99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:06:27.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T15:06:27.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:06:27.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:06:27.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:06:27.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:06:27.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:27.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:06:27.925 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:06:27.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:27.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:06:28.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:28.206 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:28.206+0000 7f6de3d99640 -1 Falling back to public interface 2024-08-19T15:06:29.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:29.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:29.196 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:06:29.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:06:29.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:29.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:06:29.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:30.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:30.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:30.480 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:06:30.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:06:30.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:30.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:06:30.628 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:30.627+0000 7f6de3d99640 -1 osd.1 0 log_to_monitors true 2024-08-19T15:06:30.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:31.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:31.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:31.751 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:06:31.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:06:31.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:31.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:06:32.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:33.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:33.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:33.026 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:06:33.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:06:33.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:33.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:06:33.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:34.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:34.302 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:06:34.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:34.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:06:34.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:34.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:06:34.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:35.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:35.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:35.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:06:35.682 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:06:35.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:35.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:06:35.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:36.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:36.957 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:06:36.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:36.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:06:36.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:36.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:06:37.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:38.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:38.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:38.235 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:06:38.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:06:38.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:38.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:06:38.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:39.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:39.513 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:06:39.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:39.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:06:39.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:39.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:06:39.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:40.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:40.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:40.783 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:06:40.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:06:40.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:40.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:06:41.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:42.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:42.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:42.057 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:06:42.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:06:42.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:42.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:06:42.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:43.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:43.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:43.587 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:06:43.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:06:43.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:43.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:06:43.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:44.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:44.990 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:06:44.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:44.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:06:44.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:44.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:06:45.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:46.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:46.451 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:06:46.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:46.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:06:46.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:46.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:06:46.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:47.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:47.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:47.735 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:06:47.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:06:47.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:47.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:06:48.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:49.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:49.272 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:06:49.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:49.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:06:49.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:49.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:06:49.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:50.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:50.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:50.794 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:06:50.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:06:50.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:50.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:06:51.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:06:52.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:06:52.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:06:52.568 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:06:52.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:06:52.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:06:52.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:06:52.885 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3766873636,v1:127.0.0.1:6811/3766873636] [v2:127.0.0.1:6812/3766873636,v1:127.0.0.1:6813/3766873636] exists,up d4a09a65-a9c7-4c80-8b37-aa2cb0bd38f9 2024-08-19T15:06:52.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:06:52.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:06:52.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:06:52.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-08-19T15:06:52.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-08-19T15:06:52.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:06:52.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:06:52.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T15:06:52.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:06:52.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:06:52.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:06:52.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:06:52.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:06:52.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:06:52.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T15:06:52.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T15:06:52.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:06:52.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:06:52.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:06:52.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:06:52.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:06:52.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:06:52.892 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:06:52.892 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:06:52.892 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:06:52.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:06:52.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:06:52.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:06:52.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:06:52.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:06:52.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:06:52.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:06:52.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:06:52.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:06:52.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:06:52.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:06:52.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:06:52.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:06:52.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T15:06:52.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:06:52.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=64998403-6f5d-462f-8b09-ed0c84bd6f6f 2024-08-19T15:06:52.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 64998403-6f5d-462f-8b09-ed0c84bd6f6f' 2024-08-19T15:06:52.893 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 64998403-6f5d-462f-8b09-ed0c84bd6f6f 2024-08-19T15:06:52.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:06:52.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCMX8NmKLIbNhAAc5Tj4+NHAo5bmIrIAUoHHg== 2024-08-19T15:06:52.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCMX8NmKLIbNhAAc5Tj4+NHAo5bmIrIAUoHHg=="}' 2024-08-19T15:06:52.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 64998403-6f5d-462f-8b09-ed0c84bd6f6f -i td/osd-scrub-repair/2/new.json 2024-08-19T15:06:53.227 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:06:53.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-08-19T15:06:53.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCMX8NmKLIbNhAAc5Tj4+NHAo5bmIrIAUoHHg== --osd-uuid 64998403-6f5d-462f-8b09-ed0c84bd6f6f 2024-08-19T15:06:53.268 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:53.267+0000 7f35b2b18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:06:53.303 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:53.302+0000 7f35b2b18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:06:53.304 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:53.303+0000 7f35b2b18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:06:53.304 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:53.304+0000 7f35b2b18640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:06:53.305 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:53.304+0000 7f35b2b18640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:06:53.305 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:53.304+0000 7f35b2b18640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:06:53.378 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:53.378+0000 7f35b2b18640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-08-19T15:06:59.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-08-19T15:06:59.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:06:59.438 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T15:06:59.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T15:06:59.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:06:59.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T15:06:59.793 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T15:06:59.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:06:59.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:06:59.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:06:59.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:06:59.819 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:59.819+0000 7fb374bba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:06:59.822 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:59.821+0000 7fb374bba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:06:59.823 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:06:59.822+0000 7fb374bba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:07:00.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T15:07:00.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:07:00.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T15:07:00.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:07:00.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:07:00.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:00.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:07:00.115 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:07:00.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:00.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:07:00.337 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:07:00.336+0000 7fb374bba640 -1 Falling back to public interface 2024-08-19T15:07:00.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:01.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:01.386 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:07:01.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:01.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:07:01.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:01.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:07:01.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:02.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:02.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:02.664 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:07:02.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:07:02.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:02.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:07:02.679 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:07:02.678+0000 7fb374bba640 -1 osd.2 0 log_to_monitors true 2024-08-19T15:07:02.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:03.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:03.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:03.980 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:07:03.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:07:03.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:03.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:07:04.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:05.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:05.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:05.268 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:07:05.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:07:05.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:05.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:07:05.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:06.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:06.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:06.612 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:07:06.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:07:06.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:06.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:07:06.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:07.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:07.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:07.889 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:07:07.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:07:07.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:07.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:07:08.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:09.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:09.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:09.165 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:07:09.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:07:09.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:09.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:07:09.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:10.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:10.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:10.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:07:10.443 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:07:10.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:10.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:07:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:11.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:11.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:11.717 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:07:11.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:07:11.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:11.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:07:11.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:12.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:12.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:12.996 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:07:12.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:07:12.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:12.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:07:13.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:14.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:14.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:14.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:07:14.273 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:07:14.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:14.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:07:15.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:16.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:16.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:16.081 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:07:16.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:07:16.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:16.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:07:16.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:17.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:17.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:17.850 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:07:17.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:07:17.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:17.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:07:18.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:19.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:19.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:19.643 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:07:19.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:07:19.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:19.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:07:20.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:21.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:21.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:21.203 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:07:21.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:07:21.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:21.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:07:21.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:22.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:22.765 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:07:22.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:22.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:07:22.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:22.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:07:23.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:24.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:24.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:24.488 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:07:24.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:07:24.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:24.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:07:24.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:25.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:25.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:25.775 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:07:25.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:07:25.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:25.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:07:26.045 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3551697849,v1:127.0.0.1:6819/3551697849] [v2:127.0.0.1:6820/3551697849,v1:127.0.0.1:6821/3551697849] exists,up 64998403-6f5d-462f-8b09-ed0c84bd6f6f 2024-08-19T15:07:26.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:07:26.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:07:26.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:07:26.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-08-19T15:07:26.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T15:07:26.338 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-08-19T15:07:26.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T15:07:26.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T15:07:26.714 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' created 2024-08-19T15:07:26.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:07:27.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T15:07:29.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-08-19T15:07:29.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-08-19T15:07:30.186 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'foo' created 2024-08-19T15:07:30.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:07:31.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2024-08-19T15:07:31.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-08-19T15:07:31.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-08-19T15:07:31.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-08-19T15:07:31.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-08-19T15:07:31.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-08-19T15:07:31.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-08-19T15:07:31.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-08-19T15:07:32.305 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'ecpool' created 2024-08-19T15:07:32.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:07:33.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-08-19T15:07:33.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-08-19T15:07:33.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:07:33.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:07:33.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:07:33.319 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:07:33.319 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:07:33.320 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:07:33.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:07:33.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:07:33.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:07:33.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:07:33.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:07:33.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:07:33.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:07:33.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:07:33.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:07:33.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:07:33.694 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:07:33.694 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:07:33.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:07:33.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:07:33.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:07:33.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-08-19T15:07:33.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-08-19T15:07:33.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-08-19T15:07:33.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:07:33.831 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:07:33.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-08-19T15:07:33.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-08-19T15:07:33.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672972' 2024-08-19T15:07:33.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:07:33.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:07:34.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-08-19T15:07:34.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-08-19T15:07:34.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672972 2-64424509448' 2024-08-19T15:07:34.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:07:34.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-08-19T15:07:34.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:07:34.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:07:34.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-08-19T15:07:34.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:07:34.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-08-19T15:07:34.039 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836500 2024-08-19T15:07:34.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-08-19T15:07:34.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:07:34.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-08-19T15:07:34.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:07:34.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-08-19T15:07:34.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:07:34.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:07:34.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-08-19T15:07:34.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:07:34.314 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672972 2024-08-19T15:07:34.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-08-19T15:07:34.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-08-19T15:07:34.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:07:34.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-08-19T15:07:34.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:07:34.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-08-19T15:07:34.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:07:34.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:07:34.591 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-08-19T15:07:34.591 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:07:34.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-08-19T15:07:34.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-08-19T15:07:34.592 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509448 2024-08-19T15:07:34.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:07:34.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-08-19T15:07:34.858 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:07:34.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:07:34.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:07:35.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:07:35.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:07:35.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:07:35.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:07:35.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:07:35.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:07:35.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:07:35.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:07:35.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:07:35.456 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:07:35.456 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:07:35.456 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:07:35.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:07:35.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:07:35.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:07:35.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-08-19T15:07:35.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-08-19T15:07:35.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:07:35.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:07:35.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:07:35.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:07:35.788 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:07:35.788 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:07:35.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:07:35.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:07:35.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:07:35.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:07:35.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:07:35.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:07:35.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:07:35.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:07:35.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:07:36.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:07:36.148 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:07:36.148 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:07:36.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:07:36.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:07:36.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:07:36.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-08-19T15:07:36.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-08-19T15:07:36.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-08-19T15:07:36.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:07:36.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:07:36.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-08-19T15:07:36.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-08-19T15:07:36.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672973' 2024-08-19T15:07:36.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:07:36.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:07:36.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-08-19T15:07:36.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-08-19T15:07:36.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672973 2-64424509450' 2024-08-19T15:07:36.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:07:36.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-08-19T15:07:36.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:07:36.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:07:36.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-08-19T15:07:36.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:07:36.472 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836502 2024-08-19T15:07:36.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-08-19T15:07:36.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-08-19T15:07:36.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:07:36.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-08-19T15:07:36.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:07:37.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:07:37.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:07:38.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-08-19T15:07:38.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:07:39.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:07:39.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:07:39.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-08-19T15:07:39.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:07:39.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-08-19T15:07:39.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:07:39.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:07:39.290 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-08-19T15:07:39.290 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:07:39.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-08-19T15:07:39.291 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672973 2024-08-19T15:07:39.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-08-19T15:07:39.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:07:39.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-08-19T15:07:39.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:07:39.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-08-19T15:07:39.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:07:39.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:07:39.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-08-19T15:07:39.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:07:39.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-08-19T15:07:39.560 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509450 2024-08-19T15:07:39.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-08-19T15:07:39.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:07:39.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-08-19T15:07:39.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:07:39.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:07:39.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:07:40.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:07:40.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:07:40.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:07:40.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:07:40.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:07:40.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:07:40.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:07:40.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:07:40.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:07:40.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:07:40.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:07:40.466 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:07:40.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:07:40.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:07:40.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:07:40.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-08-19T15:07:40.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-08-19T15:07:40.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-08-19T15:07:40.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-08-19T15:07:40.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:07:40.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:07:40.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-08-19T15:07:40.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:07:40.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:07:40.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:07:41.154 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:07:41.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:07:41.581 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:07:41.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:07:41.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:07:41.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:07:41.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-08-19T15:07:41.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-08-19T15:07:41.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-08-19T15:07:41.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-08-19T15:07:41.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-08-19T15:07:41.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:07:41.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:07:41.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:07:41.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:07:41.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:07:41.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:07:41.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:07:41.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:07:41.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:07:41.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:07:41.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:07:41.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:07:41.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:07:41.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:07:41.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:07:41.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:07:42.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:07:42.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:07:42.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:07:42.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:07:42.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:07:42.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:07:42.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:07:42.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:07:44.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:07:44.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:07:44.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:07:44.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:07:44.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:07:44.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:07:44.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:07:44.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:07:44.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:07:44.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:07:44.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:07:44.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:07:44.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:07:44.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:07:44.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:07:44.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:07:44.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:07:44.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:07:44.291 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:07:44.291 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:07:44.291 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:07:44.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:07:44.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:07:44.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:07:44.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:07:44.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:07:44.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:07:44.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:07:44.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:07:44.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:07:44.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:07:44.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:07:44.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:07:44.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:07:44.292 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:07:44.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:07:44.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:07:44.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:07:44.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:07:44.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:07:44.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:07:44.320 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:07:44.319+0000 7ff437bab640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:07:44.322 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:07:44.321+0000 7ff437bab640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:07:44.323 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:07:44.322+0000 7ff437bab640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:07:44.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:07:44.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:07:44.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:07:44.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:07:44.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:07:44.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:44.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:07:44.571 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:07:44.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:44.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:07:44.842 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:07:44.841+0000 7ff437bab640 -1 Falling back to public interface 2024-08-19T15:07:44.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:45.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:45.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:45.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:07:45.873 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:07:45.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:45.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:07:46.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:47.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:47.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:47.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:07:47.150 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:07:47.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:47.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:07:47.180 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:07:47.179+0000 7ff437bab640 -1 osd.1 35 log_to_monitors true 2024-08-19T15:07:47.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:48.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:48.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:48.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:07:48.424 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:07:48.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:48.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:07:48.704 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/3580358001,v1:127.0.0.1:6811/3580358001] [v2:127.0.0.1:6812/3580358001,v1:127.0.0.1:6813/3580358001] exists,up d4a09a65-a9c7-4c80-8b37-aa2cb0bd38f9 2024-08-19T15:07:48.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:07:48.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:07:48.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:07:48.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:07:48.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:07:48.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:07:48.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:07:48.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:07:48.705 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:07:48.705 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:07:48.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:07:48.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:07:48.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:07:48.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:07:48.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:07:48.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:07:48.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:07:48.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:07:48.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:07:49.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:07:49.069 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:07:49.070 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:07:49.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:07:49.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:07:49.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:07:49.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-08-19T15:07:49.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-08-19T15:07:49.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-08-19T15:07:49.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:07:49.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:07:49.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-08-19T15:07:49.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-08-19T15:07:49.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-171798691842' 2024-08-19T15:07:49.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:07:49.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:07:49.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-08-19T15:07:49.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-08-19T15:07:49.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-171798691842 2-64424509453' 2024-08-19T15:07:49.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:07:49.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-08-19T15:07:49.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:07:49.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:07:49.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-08-19T15:07:49.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:07:49.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-08-19T15:07:49.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-08-19T15:07:49.382 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836505 2024-08-19T15:07:49.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:07:49.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-08-19T15:07:49.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:07:50.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:07:50.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:07:50.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-08-19T15:07:50.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:07:50.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691842 2024-08-19T15:07:50.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:07:50.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:07:50.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691842 2024-08-19T15:07:50.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:07:50.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-08-19T15:07:50.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-08-19T15:07:50.927 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 171798691842 2024-08-19T15:07:50.928 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:07:51.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-08-19T15:07:51.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:07:51.195 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-08-19T15:07:51.195 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:07:51.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:07:51.196 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-08-19T15:07:51.196 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:07:51.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-08-19T15:07:51.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-08-19T15:07:51.197 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509453 2024-08-19T15:07:51.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:07:51.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-08-19T15:07:51.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:07:51.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:07:51.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:07:51.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:07:51.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:07:51.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:07:51.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:07:51.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:07:51.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:07:51.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:07:51.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:07:52.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:07:52.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:07:52.058 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:07:52.058 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:07:52.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:07:52.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:07:52.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:07:52.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-08-19T15:07:52.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-08-19T15:07:52.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-08-19T15:07:52.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:07:52.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:07:52.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-08-19T15:07:52.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:07:52.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:07:52.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:07:52.755 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:07:52.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:07:53.274 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:07:53.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:07:53.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:07:53.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:07:53.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-08-19T15:07:53.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-08-19T15:07:53.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-08-19T15:07:53.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-08-19T15:07:53.481 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records in 2024-08-19T15:07:53.481 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records out 2024-08-19T15:07:53.481 INFO:tasks.workunit.client.0.smithi102.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 6.1152e-05 s, 33.5 MB/s 2024-08-19T15:07:53.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:07:53.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:07:53.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:07:53.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:07:53.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:07:53.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:07:53.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:07:53.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:07:53.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:07:53.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:07:53.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:07:53.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:07:53.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:07:53.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:07:53.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:07:53.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:07:53.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:07:53.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:07:53.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:07:53.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:07:53.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:07:53.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:07:53.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:07:53.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:07:55.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:07:55.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:07:55.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:07:55.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:07:55.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:07:55.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:07:55.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:07:55.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:07:55.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:07:55.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:07:55.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:07:55.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:07:55.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:07:55.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:07:55.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:07:55.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:07:55.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:07:55.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:07:55.876 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:07:55.876 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:07:55.876 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:07:55.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:07:55.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:07:55.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:07:55.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:07:55.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:07:55.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:07:55.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:07:55.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:07:55.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:07:55.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:07:55.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:07:55.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:07:55.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:07:55.877 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:07:55.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:07:55.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:07:55.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:07:55.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:07:55.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:07:55.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:07:55.904 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:07:55.903+0000 7fa026f4a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:07:55.907 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:07:55.905+0000 7fa026f4a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:07:55.909 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:07:55.907+0000 7fa026f4a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:07:56.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:07:56.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:07:56.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:07:56.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:07:56.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:07:56.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:56.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:07:56.158 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:07:56.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:56.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:07:56.425 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:07:56.423+0000 7fa026f4a640 -1 Falling back to public interface 2024-08-19T15:07:56.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:57.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:57.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:57.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:07:57.466 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:07:57.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:57.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:07:57.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:07:58.683 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:07:58.681+0000 7fa026f4a640 -1 osd.0 44 log_to_monitors true 2024-08-19T15:07:58.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:07:58.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:07:58.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:07:58.739 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:07:58.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:07:58.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:07:59.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:08:00.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:08:00.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:00.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:08:00.069 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:08:00.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:00.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:08:00.505 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 49 up_thru 49 down_at 45 last_clean_interval [5,44) [v2:127.0.0.1:6802/2311201319,v1:127.0.0.1:6803/2311201319] [v2:127.0.0.1:6804/2311201319,v1:127.0.0.1:6805/2311201319] exists,up 835f901e-853d-4440-b3ef-eda73796c506 2024-08-19T15:08:00.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:08:00.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:08:00.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:08:00.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:08:00.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:08:00.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:08:00.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:08:00.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:08:00.505 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:08:00.505 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:08:00.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:08:00.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:08:00.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:08:00.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:08:00.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:08:00.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:08:00.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:08:00.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:08:00.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:08:00.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:08:00.865 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:08:00.865 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:08:00.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:08:00.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:08:00.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:08:00.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397506 2024-08-19T15:08:00.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397506 2024-08-19T15:08:00.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397506' 2024-08-19T15:08:00.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:08:00.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:08:01.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-08-19T15:08:01.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-08-19T15:08:01.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397506 1-171798691845' 2024-08-19T15:08:01.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:08:01.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:08:01.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-08-19T15:08:01.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-08-19T15:08:01.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397506 1-171798691845 2-64424509457' 2024-08-19T15:08:01.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:08:01.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-210453397506 2024-08-19T15:08:01.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:08:01.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:08:01.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-210453397506 2024-08-19T15:08:01.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:08:01.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397506 2024-08-19T15:08:01.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 210453397506' 2024-08-19T15:08:01.176 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 210453397506 2024-08-19T15:08:01.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:08:01.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 210453397506 2024-08-19T15:08:01.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:08:02.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:08:02.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:08:02.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397506 -lt 210453397506 2024-08-19T15:08:02.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:08:02.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691845 2024-08-19T15:08:02.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:08:02.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:08:02.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691845 2024-08-19T15:08:02.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:08:02.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-08-19T15:08:02.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-08-19T15:08:02.722 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 171798691845 2024-08-19T15:08:02.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:08:03.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-08-19T15:08:03.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:08:03.047 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-08-19T15:08:03.047 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:08:03.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:08:03.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-08-19T15:08:03.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:08:03.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-08-19T15:08:03.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-08-19T15:08:03.048 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509457 2024-08-19T15:08:03.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:08:03.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-08-19T15:08:03.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:08:03.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:08:03.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:08:03.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:08:03.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:08:03.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:08:03.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:08:03.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:08:03.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:08:03.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:08:03.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:08:03.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:08:03.929 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:08:03.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:08:03.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:08:04.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:08:04.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:08:04.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:08:04.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-08-19T15:08:04.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-08-19T15:08:04.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-08-19T15:08:04.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:08:04.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:08:04.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-08-19T15:08:04.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:08:04.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:08:04.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:08:04.658 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:08:04.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:08:05.169 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:08:05.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:08:05.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:08:05.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:08:05.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-08-19T15:08:05.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-08-19T15:08:05.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-08-19T15:08:05.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-08-19T15:08:05.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:08:05.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:08:05.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:08:05.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:08:05.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-08-19T15:08:05.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:08:05.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:08:05.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:08:05.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:08:05.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:08:05.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:08:05.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:08:05.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:08:05.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:08:05.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:08:05.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:08:05.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-08-19T15:08:05.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:08:05.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:08:05.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:08:05.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:08:05.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:08:05.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-08-19T15:08:07.192 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-08-19T15:08:07.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:08:07.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:08:07.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:08:07.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:08:07.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:08:07.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:08:07.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:08:07.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:08:07.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:08:07.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:08:07.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:08:07.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:08:07.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:08:07.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:08:07.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:08:07.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:08:07.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:08:07.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:08:07.713 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:08:07.713 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:08:07.713 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:08:07.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:08:07.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:08:07.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:08:07.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:08:07.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:08:07.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:08:07.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:08:07.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:08:07.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:08:07.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:08:07.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:08:07.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:08:07.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:08:07.714 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:08:07.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:08:07.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:08:07.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:08:07.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:08:07.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:08:07.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:08:07.741 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:07.740+0000 7f6f0d23f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:08:07.744 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:07.743+0000 7f6f0d23f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:08:07.745 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:07.744+0000 7f6f0d23f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:08:07.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:08:07.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:08:07.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:08:07.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:08:07.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:08:07.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:07.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:08:07.988 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:08:07.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:07.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:08:08.262 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:08.261+0000 7f6f0d23f640 -1 Falling back to public interface 2024-08-19T15:08:08.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:08:09.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:08:09.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:09.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:08:09.282 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:08:09.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:09.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:08:09.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:08:10.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:08:10.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:10.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:08:10.558 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:08:10.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:10.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:08:10.657 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:10.656+0000 7f6f0d23f640 -1 osd.1 53 log_to_monitors true 2024-08-19T15:08:10.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:08:11.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:08:11.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:11.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:08:11.845 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:08:11.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:11.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:08:12.286 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 54 last_clean_interval [40,53) [v2:127.0.0.1:6810/2968617408,v1:127.0.0.1:6811/2968617408] [v2:127.0.0.1:6812/2968617408,v1:127.0.0.1:6813/2968617408] exists,up d4a09a65-a9c7-4c80-8b37-aa2cb0bd38f9 2024-08-19T15:08:12.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:08:12.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:08:12.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:08:12.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:08:12.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:08:12.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:08:12.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:08:12.287 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:08:12.287 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:08:12.287 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:08:12.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:08:12.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:08:12.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:08:12.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:08:12.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:08:12.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:08:12.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:08:12.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:08:12.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:08:12.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:08:12.664 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:08:12.664 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:08:12.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:08:12.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:08:12.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:08:12.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397509 2024-08-19T15:08:12.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397509 2024-08-19T15:08:12.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397509' 2024-08-19T15:08:12.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:08:12.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:08:12.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103170 2024-08-19T15:08:12.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103170 2024-08-19T15:08:12.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397509 1-249108103170' 2024-08-19T15:08:12.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:08:12.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:08:12.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-08-19T15:08:12.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-08-19T15:08:12.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397509 1-249108103170 2-64424509460' 2024-08-19T15:08:12.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:08:12.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-210453397509 2024-08-19T15:08:12.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:08:12.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:08:12.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-210453397509 2024-08-19T15:08:12.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:08:12.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397509 2024-08-19T15:08:12.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 210453397509' 2024-08-19T15:08:12.974 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 210453397509 2024-08-19T15:08:12.974 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:08:13.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397508 -lt 210453397509 2024-08-19T15:08:13.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:08:14.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:08:14.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:08:14.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397510 -lt 210453397509 2024-08-19T15:08:14.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:08:14.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103170 2024-08-19T15:08:14.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:08:14.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:08:14.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103170 2024-08-19T15:08:14.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:08:14.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103170 2024-08-19T15:08:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103170' 2024-08-19T15:08:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 249108103170 2024-08-19T15:08:14.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:08:14.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103170 -lt 249108103170 2024-08-19T15:08:14.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:08:14.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-08-19T15:08:14.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:08:14.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:08:14.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-08-19T15:08:14.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:08:14.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-08-19T15:08:14.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-08-19T15:08:14.790 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509460 2024-08-19T15:08:14.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:08:15.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-08-19T15:08:15.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:08:15.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:08:15.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:08:15.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:08:15.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:08:15.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:08:15.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:08:15.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:08:15.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:08:15.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:08:15.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:08:15.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:08:15.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:08:15.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:08:15.695 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:08:16.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:08:16.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:08:16.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:08:16.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-08-19T15:08:16.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-08-19T15:08:16.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-08-19T15:08:16.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:08:16.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:08:16.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-08-19T15:08:16.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:08:16.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:08:16.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:08:16.508 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:08:16.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:08:16.884 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:08:16.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:08:16.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:08:16.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:08:17.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-08-19T15:08:17.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-08-19T15:08:17.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-08-19T15:08:17.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-08-19T15:08:17.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-08-19T15:08:17.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-08-19T15:08:17.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-08-19T15:08:17.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:08:17.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:08:17.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:08:17.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:08:17.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-08-19T15:08:17.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:08:17.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:08:17.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:08:17.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:08:17.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:08:17.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:08:17.494 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:08:17.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:08:17.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:08:17.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:08:17.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:08:17.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-08-19T15:08:17.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:08:17.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:08:17.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:08:17.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:08:17.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:08:17.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-08-19T15:08:19.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:08:19.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:08:19.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:08:19.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:08:19.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:08:19.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:08:19.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:08:19.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:08:19.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:08:19.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:08:19.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:08:19.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:08:19.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:08:19.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:08:19.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:08:19.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:08:19.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:08:19.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:08:19.797 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:08:19.798 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:08:19.798 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:08:19.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:08:19.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:08:19.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:08:19.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:08:19.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:08:19.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:08:19.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:08:19.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:08:19.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:08:19.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:08:19.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:08:19.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:08:19.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:08:19.799 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:08:19.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:08:19.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:08:19.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:08:19.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:08:19.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:08:19.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:08:19.828 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:19.827+0000 7f2a74611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:08:19.830 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:19.829+0000 7f2a74611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:08:19.831 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:19.830+0000 7f2a74611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:08:20.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:08:20.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:08:20.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:08:20.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:08:20.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:08:20.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:20.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:08:20.083 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:08:20.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:20.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:08:20.348 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:20.347+0000 7f2a74611640 -1 Falling back to public interface 2024-08-19T15:08:20.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:08:21.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:08:21.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:21.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:08:21.393 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:08:21.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:21.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:08:21.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:08:22.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:08:22.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:22.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:08:22.668 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:08:22.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:22.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:08:22.754 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:22.753+0000 7f2a74611640 -1 osd.0 62 log_to_monitors true 2024-08-19T15:08:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:08:23.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:08:23.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:23.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:08:23.956 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:08:23.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:23.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:08:24.296 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 67 up_thru 67 down_at 63 last_clean_interval [49,62) [v2:127.0.0.1:6802/1497965084,v1:127.0.0.1:6803/1497965084] [v2:127.0.0.1:6804/1497965084,v1:127.0.0.1:6805/1497965084] exists,up 835f901e-853d-4440-b3ef-eda73796c506 2024-08-19T15:08:24.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:08:24.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:08:24.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:08:24.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:08:24.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:08:24.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:08:24.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:08:24.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:08:24.297 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:08:24.297 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:08:24.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:08:24.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:08:24.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:08:24.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:08:24.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:08:24.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:08:24.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:08:24.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:08:24.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:08:24.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:08:24.753 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:08:24.753 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:08:24.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:08:24.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:08:24.753 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:08:24.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-08-19T15:08:24.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-08-19T15:08:24.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808834' 2024-08-19T15:08:24.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:08:24.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:08:24.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103173 2024-08-19T15:08:24.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103173 2024-08-19T15:08:24.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808834 1-249108103173' 2024-08-19T15:08:24.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:08:24.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:08:25.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-08-19T15:08:25.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-08-19T15:08:25.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808834 1-249108103173 2-64424509463' 2024-08-19T15:08:25.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:08:25.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808834 2024-08-19T15:08:25.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:08:25.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:08:25.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808834 2024-08-19T15:08:25.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:08:25.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-08-19T15:08:25.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808834' 2024-08-19T15:08:25.072 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 287762808834 2024-08-19T15:08:25.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:08:25.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 287762808834 2024-08-19T15:08:25.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:08:26.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:08:26.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:08:26.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808834 -lt 287762808834 2024-08-19T15:08:26.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:08:26.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103173 2024-08-19T15:08:26.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:08:26.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:08:26.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103173 2024-08-19T15:08:26.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:08:26.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103173 2024-08-19T15:08:26.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103173' 2024-08-19T15:08:26.613 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 249108103173 2024-08-19T15:08:26.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:08:26.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103174 -lt 249108103173 2024-08-19T15:08:26.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:08:26.880 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-08-19T15:08:26.880 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:08:26.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:08:26.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-08-19T15:08:26.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:08:26.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-08-19T15:08:26.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-08-19T15:08:26.883 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509463 2024-08-19T15:08:26.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:08:27.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-08-19T15:08:27.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:08:27.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:08:27.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:08:27.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:08:27.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:08:27.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:08:27.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:08:27.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:08:27.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:08:27.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:08:27.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:08:27.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:08:27.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:08:27.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:08:27.750 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:08:28.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:08:28.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:08:28.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:08:28.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-08-19T15:08:28.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:08:28.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:08:28.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-08-19T15:08:28.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:08:28.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-08-19T15:08:28.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:08:28.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:08:28.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-08-19T15:08:28.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:08:28.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-08-19T15:08:28.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:08:28.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:08:28.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:08:28.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:08:28.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:08:28.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:08:28.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-08-19T15:08:28.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:08:28.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:08:28.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-08-19T15:08:28.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:08:28.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:08:28.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-08-19T15:08:30.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-08-19T15:08:30.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:08:30.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:08:30.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-19T15:08:30.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:08:30.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:08:30.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:08:30.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:08:30.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:08:30.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:08:30.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T15:08:30.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T15:08:30.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:08:30.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:08:30.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:08:30.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:08:30.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:08:30.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:08:30.349 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:08:30.349 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:08:30.349 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:08:30.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:08:30.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:08:30.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:08:30.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:08:30.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:08:30.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:08:30.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:08:30.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:08:30.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:08:30.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:08:30.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:08:30.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T15:08:30.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-19T15:08:30.351 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.2 2024-08-19T15:08:30.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:08:30.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-08-19T15:08:30.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-19T15:08:30.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:08:30.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:08:30.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:08:30.379 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:30.378+0000 7fef1a820640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:08:30.381 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:30.381+0000 7fef1a820640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:08:30.383 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:30.382+0000 7fef1a820640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:08:30.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-19T15:08:30.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:08:30.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T15:08:30.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:08:30.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:08:30.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:30.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:08:30.632 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:08:30.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:30.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:08:30.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:08:30.900 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:30.900+0000 7fef1a820640 -1 Falling back to public interface 2024-08-19T15:08:31.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:08:31.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:31.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:08:31.901 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:08:31.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:31.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:08:32.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:08:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:08:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:08:33.178 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:08:33.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:33.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:08:33.253 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:33.252+0000 7fef1a820640 -1 osd.2 69 log_to_monitors true 2024-08-19T15:08:33.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:08:34.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:08:34.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:34.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:08:34.459 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:08:34.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:34.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:08:34.865 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 up in weight 1 up_from 74 up_thru 74 down_at 70 last_clean_interval [15,69) [v2:127.0.0.1:6818/1822335947,v1:127.0.0.1:6819/1822335947] [v2:127.0.0.1:6820/1822335947,v1:127.0.0.1:6821/1822335947] exists,up 64998403-6f5d-462f-8b09-ed0c84bd6f6f 2024-08-19T15:08:34.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:08:34.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:08:34.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:08:34.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:08:34.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:08:34.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:08:34.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:08:34.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:08:34.866 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:08:34.866 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:08:34.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:08:34.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:08:34.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:08:34.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:08:34.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:08:34.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:08:34.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:08:34.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:08:34.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:08:35.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:08:35.250 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:08:35.250 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:08:35.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:08:35.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:08:35.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:08:35.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808837 2024-08-19T15:08:35.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808837 2024-08-19T15:08:35.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808837' 2024-08-19T15:08:35.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:08:35.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:08:35.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103176 2024-08-19T15:08:35.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103176 2024-08-19T15:08:35.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808837 1-249108103176' 2024-08-19T15:08:35.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:08:35.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:08:35.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-08-19T15:08:35.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-08-19T15:08:35.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808837 1-249108103176 2-317827579906' 2024-08-19T15:08:35.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:08:35.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808837 2024-08-19T15:08:35.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:08:35.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:08:35.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808837 2024-08-19T15:08:35.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:08:35.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808837 2024-08-19T15:08:35.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808837' 2024-08-19T15:08:35.560 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 287762808837 2024-08-19T15:08:35.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:08:35.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808836 -lt 287762808837 2024-08-19T15:08:35.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:08:36.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:08:36.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:08:37.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808837 -lt 287762808837 2024-08-19T15:08:37.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:08:37.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103176 2024-08-19T15:08:37.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:08:37.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:08:37.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103176 2024-08-19T15:08:37.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:08:37.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103176 2024-08-19T15:08:37.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103176' 2024-08-19T15:08:37.105 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 249108103176 2024-08-19T15:08:37.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:08:37.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103177 -lt 249108103176 2024-08-19T15:08:37.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:08:37.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579906 2024-08-19T15:08:37.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:08:37.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:08:37.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579906 2024-08-19T15:08:37.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:08:37.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-08-19T15:08:37.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579906' 2024-08-19T15:08:37.374 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 317827579906 2024-08-19T15:08:37.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:08:37.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579906 -lt 317827579906 2024-08-19T15:08:37.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:08:37.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:08:37.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:08:37.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:08:37.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:08:37.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:08:37.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:08:37.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:08:37.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:08:37.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:08:37.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:08:38.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:08:38.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:08:38.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:08:38.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:08:38.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:08:38.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:08:38.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:08:38.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-08-19T15:08:38.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-08-19T15:08:38.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:08:38.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:08:38.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-08-19T15:08:38.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:08:38.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-08-19T15:08:38.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:08:38.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:08:38.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-08-19T15:08:38.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:08:38.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-08-19T15:08:38.577 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:08:38.577 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:08:38.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:08:38.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:08:38.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:08:38.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:08:38.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-08-19T15:08:38.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:08:38.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:08:38.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-08-19T15:08:38.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:08:38.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:08:38.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-08-19T15:08:40.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-08-19T15:08:40.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:08:40.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:08:40.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-19T15:08:40.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:08:40.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:08:40.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:08:40.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:08:40.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:08:40.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:08:40.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T15:08:40.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T15:08:40.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:08:40.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:08:40.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:08:40.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:08:40.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:08:40.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:08:40.994 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:08:40.994 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:08:40.994 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:08:40.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:08:40.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:08:40.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:08:40.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:08:40.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:08:40.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:08:40.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:08:40.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:08:40.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:08:40.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:08:40.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:08:40.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T15:08:40.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-19T15:08:40.996 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.2 2024-08-19T15:08:40.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:08:40.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-08-19T15:08:40.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-19T15:08:40.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:08:40.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:08:40.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:08:41.023 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:41.022+0000 7fb2bdc18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:08:41.026 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:41.025+0000 7fb2bdc18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:08:41.027 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:41.026+0000 7fb2bdc18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:08:41.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-19T15:08:41.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:08:41.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T15:08:41.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:08:41.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:08:41.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:41.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:08:41.276 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:08:41.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:41.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:08:41.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:08:41.553 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:41.552+0000 7fb2bdc18640 -1 Falling back to public interface 2024-08-19T15:08:42.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:08:42.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:42.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:08:42.542 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:08:42.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:42.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:08:42.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:08:43.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:08:43.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:43.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:08:43.817 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:08:43.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:43.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:08:43.848 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:43.847+0000 7fb2bdc18640 -1 osd.2 77 log_to_monitors true 2024-08-19T15:08:44.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:08:45.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:08:45.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:45.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:08:45.204 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:08:45.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:45.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:08:45.492 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 up in weight 1 up_from 82 up_thru 82 down_at 78 last_clean_interval [74,77) [v2:127.0.0.1:6818/4288708200,v1:127.0.0.1:6819/4288708200] [v2:127.0.0.1:6820/4288708200,v1:127.0.0.1:6821/4288708200] exists,up 64998403-6f5d-462f-8b09-ed0c84bd6f6f 2024-08-19T15:08:45.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:08:45.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:08:45.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:08:45.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:08:45.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:08:45.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:08:45.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:08:45.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:08:45.493 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:08:45.493 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:08:45.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:08:45.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:08:45.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:08:45.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:08:45.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:08:45.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:08:45.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:08:45.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:08:45.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:08:45.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:08:45.919 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:08:45.920 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:08:45.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:08:45.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:08:45.920 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:08:46.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808840 2024-08-19T15:08:46.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808840 2024-08-19T15:08:46.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808840' 2024-08-19T15:08:46.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:08:46.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:08:46.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103180 2024-08-19T15:08:46.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103180 2024-08-19T15:08:46.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808840 1-249108103180' 2024-08-19T15:08:46.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:08:46.126 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:08:46.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-08-19T15:08:46.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-08-19T15:08:46.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808840 1-249108103180 2-352187318274' 2024-08-19T15:08:46.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:08:46.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808840 2024-08-19T15:08:46.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:08:46.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:08:46.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808840 2024-08-19T15:08:46.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:08:46.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808840 2024-08-19T15:08:46.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808840' 2024-08-19T15:08:46.233 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 287762808840 2024-08-19T15:08:46.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:08:46.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808840 -lt 287762808840 2024-08-19T15:08:46.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:08:46.497 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103180 2024-08-19T15:08:46.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:08:46.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:08:46.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103180 2024-08-19T15:08:46.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:08:46.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103180 2024-08-19T15:08:46.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103180' 2024-08-19T15:08:46.500 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 249108103180 2024-08-19T15:08:46.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:08:46.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103179 -lt 249108103180 2024-08-19T15:08:46.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:08:47.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:08:47.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:08:48.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103179 -lt 249108103180 2024-08-19T15:08:48.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:08:49.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:08:49.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:08:49.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103180 -lt 249108103180 2024-08-19T15:08:49.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:08:49.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318274 2024-08-19T15:08:49.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:08:49.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:08:49.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318274 2024-08-19T15:08:49.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:08:49.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-08-19T15:08:49.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318274' 2024-08-19T15:08:49.325 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 352187318274 2024-08-19T15:08:49.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:08:49.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318274 -lt 352187318274 2024-08-19T15:08:49.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:08:49.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:08:49.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:08:49.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:08:49.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:08:49.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:08:49.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:08:49.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:08:49.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:08:49.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:08:49.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:08:50.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:08:50.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:08:50.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:08:50.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:08:50.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:08:50.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:08:50.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:08:50.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-08-19T15:08:50.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-08-19T15:08:50.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-08-19T15:08:50.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-08-19T15:08:50.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:08:50.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:08:50.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-08-19T15:08:50.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:08:50.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:08:50.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:08:50.893 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:08:50.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:08:51.262 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:08:51.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:08:51.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:08:51.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:08:51.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-08-19T15:08:51.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-08-19T15:08:51.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-08-19T15:08:51.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-08-19T15:08:51.504 INFO:tasks.workunit.client.0.smithi102.stderr:2+0 records in 2024-08-19T15:08:51.504 INFO:tasks.workunit.client.0.smithi102.stderr:2+0 records out 2024-08-19T15:08:51.504 INFO:tasks.workunit.client.0.smithi102.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 6.8597e-05 s, 59.7 MB/s 2024-08-19T15:08:51.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:08:51.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:08:51.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:08:51.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:08:51.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:08:51.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:08:51.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:08:51.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:08:51.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:08:51.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:08:51.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:08:51.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:08:51.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:08:51.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:08:51.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:08:51.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:08:51.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:08:51.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:08:51.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:08:51.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:08:51.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:08:51.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:08:51.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:08:51.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:08:53.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:08:53.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:08:53.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:08:53.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:08:53.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:08:53.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:08:53.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:08:53.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:08:53.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:08:53.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:08:53.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:08:53.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:08:53.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:08:53.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:08:53.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:08:53.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:08:53.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:08:53.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:08:53.999 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:08:53.999 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:08:53.999 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:08:53.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:08:53.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:08:53.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:08:53.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:08:53.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:08:54.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:08:54.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:08:54.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:08:54.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:08:54.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:08:54.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:08:54.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:08:54.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:08:54.001 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:08:54.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:08:54.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:08:54.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:08:54.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:08:54.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:08:54.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:08:54.028 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:54.028+0000 7f8f6dc0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:08:54.031 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:54.030+0000 7f8f6dc0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:08:54.032 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:54.031+0000 7f8f6dc0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:08:54.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:08:54.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:08:54.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:08:54.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:08:54.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:08:54.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:54.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:08:54.281 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:08:54.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:54.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:08:54.547 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:54.546+0000 7f8f6dc0e640 -1 Falling back to public interface 2024-08-19T15:08:54.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:08:55.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:08:55.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:55.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:08:55.563 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:08:55.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:55.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:08:55.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:08:56.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:08:56.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:56.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:08:56.834 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:08:56.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:56.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:08:56.934 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:08:56.933+0000 7f8f6dc0e640 -1 osd.1 87 log_to_monitors true 2024-08-19T15:08:57.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:08:58.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:08:58.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:08:58.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:08:58.198 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:08:58.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:08:58.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:08:58.510 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 92 up_thru 92 down_at 88 last_clean_interval [58,87) [v2:127.0.0.1:6810/491737611,v1:127.0.0.1:6811/491737611] [v2:127.0.0.1:6812/491737611,v1:127.0.0.1:6813/491737611] exists,up d4a09a65-a9c7-4c80-8b37-aa2cb0bd38f9 2024-08-19T15:08:58.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:08:58.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:08:58.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:08:58.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:08:58.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:08:58.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:08:58.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:08:58.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:08:58.511 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:08:58.512 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:08:58.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:08:58.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:08:58.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:08:58.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:08:58.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:08:58.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:08:58.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:08:58.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:08:58.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:08:58.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:08:58.884 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:08:58.884 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:08:58.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:08:58.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:08:58.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:08:59.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808844 2024-08-19T15:08:59.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808844 2024-08-19T15:08:59.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808844' 2024-08-19T15:08:59.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:08:59.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:08:59.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991234 2024-08-19T15:08:59.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991234 2024-08-19T15:08:59.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808844 1-395136991234' 2024-08-19T15:08:59.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:08:59.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:08:59.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318278 2024-08-19T15:08:59.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318278 2024-08-19T15:08:59.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808844 1-395136991234 2-352187318278' 2024-08-19T15:08:59.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:08:59.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808844 2024-08-19T15:08:59.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:08:59.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:08:59.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808844 2024-08-19T15:08:59.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:08:59.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808844 2024-08-19T15:08:59.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808844' 2024-08-19T15:08:59.216 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 287762808844 2024-08-19T15:08:59.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:08:59.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808843 -lt 287762808844 2024-08-19T15:08:59.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:09:00.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:09:00.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:09:00.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808844 -lt 287762808844 2024-08-19T15:09:00.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:09:00.752 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-395136991234 2024-08-19T15:09:00.752 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:09:00.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:09:00.753 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-395136991234 2024-08-19T15:09:00.753 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:09:00.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991234 2024-08-19T15:09:00.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 395136991234' 2024-08-19T15:09:00.754 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 395136991234 2024-08-19T15:09:00.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:09:01.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991234 -lt 395136991234 2024-08-19T15:09:01.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:09:01.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318278 2024-08-19T15:09:01.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:09:01.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:09:01.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318278 2024-08-19T15:09:01.024 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:09:01.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318278 2024-08-19T15:09:01.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318278' 2024-08-19T15:09:01.025 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 352187318278 2024-08-19T15:09:01.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:09:01.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318278 -lt 352187318278 2024-08-19T15:09:01.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:09:01.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:09:01.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:09:01.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:09:01.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:09:01.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:09:01.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:09:01.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:09:01.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:09:01.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:09:01.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:09:01.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:09:01.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:09:01.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:09:01.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:09:02.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:09:02.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:09:02.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:09:02.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-08-19T15:09:02.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-08-19T15:09:02.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-08-19T15:09:02.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:09:02.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:09:02.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-08-19T15:09:02.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:09:02.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:09:02.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:09:02.577 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:09:02.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:09:03.079 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:09:03.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:09:03.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:09:03.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:09:03.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-08-19T15:09:03.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-08-19T15:09:03.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-08-19T15:09:03.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-08-19T15:09:03.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:09:03.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:09:03.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:09:03.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:09:03.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-08-19T15:09:03.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:09:03.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:09:03.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:09:03.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:09:03.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:09:03.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:09:03.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:09:03.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:09:03.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:09:03.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:09:03.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:09:03.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-08-19T15:09:03.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:09:03.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:09:03.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:09:03.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:09:03.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:09:03.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-08-19T15:09:05.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:09:05.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:09:05.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:09:05.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:09:05.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:09:05.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:09:05.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:09:05.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:09:05.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:09:05.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:09:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:09:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:09:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:09:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:09:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:09:05.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:09:05.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:09:05.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:09:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:09:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:09:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:09:05.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:09:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:09:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:09:05.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:09:05.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:09:05.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:09:05.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:09:05.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:09:05.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:09:05.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:09:05.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:09:05.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:09:05.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:09:05.614 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:09:05.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:09:05.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:09:05.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:09:05.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:09:05.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:09:05.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:09:05.641 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:05.640+0000 7fe69dc9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:09:05.644 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:05.643+0000 7fe69dc9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:09:05.645 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:05.644+0000 7fe69dc9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:09:05.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:09:05.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:09:05.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:09:05.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:09:05.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:09:05.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:09:05.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:09:05.889 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:09:05.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:09:05.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:09:06.163 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:06.162+0000 7fe69dc9c640 -1 Falling back to public interface 2024-08-19T15:09:06.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:09:07.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:09:07.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:09:07.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:09:07.200 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:09:07.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:09:07.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:09:07.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:09:08.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:09:08.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:09:08.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:09:08.471 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:09:08.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:09:08.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:09:08.691 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:08.690+0000 7fe69dc9c640 -1 osd.0 96 log_to_monitors true 2024-08-19T15:09:08.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:09:09.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:09:09.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:09:09.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:09:09.745 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:09:09.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:09:09.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:09:10.171 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 101 up_thru 101 down_at 97 last_clean_interval [67,96) [v2:127.0.0.1:6802/1573004506,v1:127.0.0.1:6803/1573004506] [v2:127.0.0.1:6804/1573004506,v1:127.0.0.1:6805/1573004506] exists,up 835f901e-853d-4440-b3ef-eda73796c506 2024-08-19T15:09:10.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:09:10.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:09:10.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:09:10.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:09:10.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:09:10.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:09:10.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:09:10.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:09:10.171 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:09:10.172 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:09:10.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:09:10.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:09:10.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:09:10.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:09:10.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:09:10.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:09:10.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:09:10.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:09:10.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:09:10.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:09:10.531 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:09:10.531 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:09:10.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:09:10.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:09:10.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:09:10.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696898 2024-08-19T15:09:10.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696898 2024-08-19T15:09:10.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696898' 2024-08-19T15:09:10.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:09:10.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:09:10.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991237 2024-08-19T15:09:10.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991237 2024-08-19T15:09:10.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696898 1-395136991237' 2024-08-19T15:09:10.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:09:10.752 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:09:10.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318281 2024-08-19T15:09:10.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318281 2024-08-19T15:09:10.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696898 1-395136991237 2-352187318281' 2024-08-19T15:09:10.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:09:10.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-433791696898 2024-08-19T15:09:10.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:09:10.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:09:10.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-433791696898 2024-08-19T15:09:10.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:09:10.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696898 2024-08-19T15:09:10.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 433791696898' 2024-08-19T15:09:10.857 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 433791696898 2024-08-19T15:09:10.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:09:11.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 433791696898 2024-08-19T15:09:11.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:09:12.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:09:12.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:09:12.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696898 -lt 433791696898 2024-08-19T15:09:12.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:09:12.397 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-395136991237 2024-08-19T15:09:12.397 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:09:12.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:09:12.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-395136991237 2024-08-19T15:09:12.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:09:12.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991237 2024-08-19T15:09:12.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 395136991237' 2024-08-19T15:09:12.400 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 395136991237 2024-08-19T15:09:12.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:09:12.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991237 -lt 395136991237 2024-08-19T15:09:12.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:09:12.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318281 2024-08-19T15:09:12.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:09:12.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:09:12.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318281 2024-08-19T15:09:12.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:09:12.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318281 2024-08-19T15:09:12.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318281' 2024-08-19T15:09:12.670 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 352187318281 2024-08-19T15:09:12.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:09:12.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318281 -lt 352187318281 2024-08-19T15:09:12.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:09:12.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:09:12.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:09:13.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:09:13.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:09:13.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:09:13.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:09:13.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:09:13.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:09:13.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:09:13.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:09:13.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:09:13.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:09:13.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:09:13.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:09:13.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:09:13.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:09:13.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:09:13.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-08-19T15:09:13.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-08-19T15:09:13.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:09:13.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:09:13.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:09:13.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:09:13.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-08-19T15:09:13.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:09:13.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:09:13.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:09:13.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:09:13.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:09:13.861 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:09:13.861 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:09:13.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:09:13.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:09:13.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:09:14.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:09:14.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-08-19T15:09:14.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:09:14.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:09:14.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:09:14.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:09:14.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:09:14.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-08-19T15:09:16.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:09:16.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:09:16.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:09:16.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:09:16.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:09:16.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:09:16.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:09:16.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:09:16.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:09:16.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:09:16.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:09:16.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:09:16.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:09:16.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:09:16.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:09:16.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:09:16.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:09:16.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:09:16.187 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:09:16.187 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:09:16.187 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:09:16.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:09:16.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:09:16.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:09:16.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:09:16.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:09:16.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:09:16.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:09:16.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:09:16.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:09:16.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:09:16.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:09:16.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:09:16.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:09:16.189 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:09:16.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:09:16.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:09:16.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:09:16.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:09:16.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:09:16.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:09:16.217 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:16.215+0000 7f0544c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:09:16.218 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:16.218+0000 7f0544c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:09:16.220 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:16.219+0000 7f0544c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:09:16.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:09:16.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:09:16.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:09:16.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:09:16.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:09:16.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:09:16.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:09:16.472 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:09:16.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:09:16.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:09:16.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:09:16.745 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:16.744+0000 7f0544c18640 -1 Falling back to public interface 2024-08-19T15:09:17.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:09:17.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:09:17.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:09:17.737 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:09:17.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:09:17.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:09:18.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:09:19.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:09:19.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:09:19.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:09:19.014 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:09:19.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:09:19.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:09:19.020 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:19.020+0000 7f0544c18640 -1 osd.1 103 log_to_monitors true 2024-08-19T15:09:19.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:09:20.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:09:20.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:09:20.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:09:20.393 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:09:20.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:09:20.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:09:20.777 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 108 up_thru 108 down_at 104 last_clean_interval [92,103) [v2:127.0.0.1:6810/1663258462,v1:127.0.0.1:6811/1663258462] [v2:127.0.0.1:6812/1663258462,v1:127.0.0.1:6813/1663258462] exists,up d4a09a65-a9c7-4c80-8b37-aa2cb0bd38f9 2024-08-19T15:09:20.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:09:20.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:09:20.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:09:20.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:09:20.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:09:20.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:09:20.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:09:20.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:09:20.778 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:09:20.778 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:09:20.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:09:20.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:09:20.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:09:20.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:09:20.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:09:20.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:09:20.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:09:20.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:09:20.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:09:21.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:09:21.133 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:09:21.133 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:09:21.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:09:21.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:09:21.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:09:21.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696901 2024-08-19T15:09:21.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696901 2024-08-19T15:09:21.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696901' 2024-08-19T15:09:21.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:09:21.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:09:21.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467970 2024-08-19T15:09:21.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467970 2024-08-19T15:09:21.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696901 1-463856467970' 2024-08-19T15:09:21.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:09:21.340 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:09:21.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318284 2024-08-19T15:09:21.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318284 2024-08-19T15:09:21.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696901 1-463856467970 2-352187318284' 2024-08-19T15:09:21.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:09:21.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-433791696901 2024-08-19T15:09:21.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:09:21.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:09:21.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-433791696901 2024-08-19T15:09:21.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:09:21.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696901 2024-08-19T15:09:21.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 433791696901' 2024-08-19T15:09:21.445 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 433791696901 2024-08-19T15:09:21.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:09:21.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696900 -lt 433791696901 2024-08-19T15:09:21.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:09:22.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:09:22.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:09:22.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696901 -lt 433791696901 2024-08-19T15:09:22.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:09:22.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-463856467970 2024-08-19T15:09:22.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:09:22.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:09:22.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-463856467970 2024-08-19T15:09:22.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:09:22.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467970 2024-08-19T15:09:22.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 463856467970' 2024-08-19T15:09:22.990 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 463856467970 2024-08-19T15:09:22.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:09:23.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467970 -lt 463856467970 2024-08-19T15:09:23.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:09:23.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318284 2024-08-19T15:09:23.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:09:23.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:09:23.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318284 2024-08-19T15:09:23.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:09:23.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318284 2024-08-19T15:09:23.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318284' 2024-08-19T15:09:23.260 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 352187318284 2024-08-19T15:09:23.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:09:23.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318284 -lt 352187318284 2024-08-19T15:09:23.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:09:23.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:09:23.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:09:23.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:09:23.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:09:23.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:09:23.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:09:23.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:09:23.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:09:23.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:09:23.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:09:24.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:09:24.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:09:24.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:09:24.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:09:24.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:09:24.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:09:24.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:09:24.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-08-19T15:09:24.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-08-19T15:09:24.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-08-19T15:09:24.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:09:24.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:09:24.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-08-19T15:09:24.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:09:24.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:09:24.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:09:24.853 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:09:24.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:09:25.364 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:09:25.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:09:25.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:09:25.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:09:25.576 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-08-19T15:09:25.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-08-19T15:09:25.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-08-19T15:09:25.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-08-19T15:09:25.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-08-19T15:09:25.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-08-19T15:09:25.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-08-19T15:09:25.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:09:25.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:09:25.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:09:25.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:09:25.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-08-19T15:09:25.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:09:25.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:09:25.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:09:25.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:09:25.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:09:25.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:09:25.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:09:25.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:09:25.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:09:25.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:09:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:09:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-08-19T15:09:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:09:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:09:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:09:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:09:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:09:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-08-19T15:09:27.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:09:27.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:09:27.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:09:27.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:09:27.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:09:27.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:09:27.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:09:27.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:09:27.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:09:27.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:09:27.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:09:27.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:09:27.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:09:27.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:09:27.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:09:27.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:09:27.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:09:27.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:09:27.790 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:09:27.790 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:09:27.790 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:09:27.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:09:27.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:09:27.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:09:27.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:09:27.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:09:27.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:09:27.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:09:27.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:09:27.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:09:27.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:09:27.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:09:27.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:09:27.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:09:27.792 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:09:27.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:09:27.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:09:27.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:09:27.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:09:27.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:09:27.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:09:27.819 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:27.818+0000 7fb9fc00c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:09:27.822 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:27.821+0000 7fb9fc00c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:09:27.823 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:27.822+0000 7fb9fc00c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:09:28.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:09:28.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:09:28.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:09:28.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:09:28.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:09:28.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:09:28.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:09:28.071 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:09:28.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:09:28.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:09:28.341 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:28.340+0000 7fb9fc00c640 -1 Falling back to public interface 2024-08-19T15:09:28.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:09:29.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:09:29.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:09:29.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:09:29.403 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:09:29.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:09:29.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:09:29.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:09:30.669 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:30.668+0000 7fb9fc00c640 -1 osd.0 112 log_to_monitors true 2024-08-19T15:09:30.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:09:30.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:09:30.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:09:30.679 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:09:30.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:09:30.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:09:31.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:09:32.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:09:32.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:09:32.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:09:32.057 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:09:32.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:09:32.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:09:32.335 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 117 up_thru 117 down_at 113 last_clean_interval [101,112) [v2:127.0.0.1:6802/1800544765,v1:127.0.0.1:6803/1800544765] [v2:127.0.0.1:6804/1800544765,v1:127.0.0.1:6805/1800544765] exists,up 835f901e-853d-4440-b3ef-eda73796c506 2024-08-19T15:09:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:09:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:09:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:09:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:09:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:09:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:09:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:09:32.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:09:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:09:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:09:32.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:09:32.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:09:32.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:09:32.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:09:32.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:09:32.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:09:32.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:09:32.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:09:32.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:09:32.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:09:32.690 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:09:32.690 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:09:32.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:09:32.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:09:32.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:09:32.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=502511173634 2024-08-19T15:09:32.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 502511173634 2024-08-19T15:09:32.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-502511173634' 2024-08-19T15:09:32.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:09:32.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:09:32.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467973 2024-08-19T15:09:32.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467973 2024-08-19T15:09:32.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-502511173634 1-463856467973' 2024-08-19T15:09:32.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:09:32.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:09:33.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318287 2024-08-19T15:09:33.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318287 2024-08-19T15:09:33.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-502511173634 1-463856467973 2-352187318287' 2024-08-19T15:09:33.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:09:33.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-502511173634 2024-08-19T15:09:33.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:09:33.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:09:33.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-502511173634 2024-08-19T15:09:33.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:09:33.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=502511173634 2024-08-19T15:09:33.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 502511173634' 2024-08-19T15:09:33.089 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 502511173634 2024-08-19T15:09:33.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:09:33.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 502511173634 2024-08-19T15:09:33.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:09:34.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:09:34.356 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:09:34.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 502511173634 -lt 502511173634 2024-08-19T15:09:34.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:09:34.631 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-463856467973 2024-08-19T15:09:34.631 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:09:34.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:09:34.633 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-463856467973 2024-08-19T15:09:34.633 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:09:34.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467973 2024-08-19T15:09:34.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 463856467973' 2024-08-19T15:09:34.634 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 463856467973 2024-08-19T15:09:34.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:09:34.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467973 -lt 463856467973 2024-08-19T15:09:34.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:09:34.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318287 2024-08-19T15:09:34.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:09:34.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:09:34.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318287 2024-08-19T15:09:34.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:09:34.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318287 2024-08-19T15:09:34.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318287' 2024-08-19T15:09:34.903 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 352187318287 2024-08-19T15:09:34.903 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:09:35.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318287 -lt 352187318287 2024-08-19T15:09:35.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:09:35.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:09:35.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:09:35.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:09:35.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:09:35.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:09:35.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:09:35.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:09:35.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:09:35.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:09:35.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:09:35.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:09:35.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:09:35.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:09:35.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:09:36.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:09:36.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:09:36.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:09:36.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-08-19T15:09:36.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:09:36.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:09:36.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:09:36.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:09:36.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-08-19T15:09:36.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:09:36.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:09:36.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:09:36.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:09:36.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:09:36.179 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:09:36.179 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:09:36.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:09:36.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:09:36.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:09:36.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:09:36.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-08-19T15:09:36.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:09:36.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:09:36.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:09:36.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:09:36.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:09:36.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-08-19T15:09:38.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:09:38.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:09:38.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:09:38.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:09:38.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:09:38.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:09:38.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:09:38.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:09:38.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:09:38.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:09:38.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:09:38.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:09:38.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:09:38.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:09:38.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:09:38.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:09:38.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:09:38.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:09:38.485 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:09:38.485 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:09:38.485 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:09:38.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:09:38.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:09:38.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:09:38.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:09:38.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:09:38.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:09:38.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:09:38.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:09:38.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:09:38.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:09:38.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:09:38.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:09:38.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:09:38.487 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:09:38.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:09:38.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:09:38.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:09:38.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:09:38.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:09:38.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:09:38.514 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:38.513+0000 7f4ef9357640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:09:38.516 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:38.515+0000 7f4ef9357640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:09:38.517 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:38.516+0000 7f4ef9357640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:09:38.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:09:38.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:09:38.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:09:38.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:09:38.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:09:38.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:09:38.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:09:38.764 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:09:38.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:09:38.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:09:39.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:09:39.041 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:39.040+0000 7f4ef9357640 -1 Falling back to public interface 2024-08-19T15:09:40.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:09:40.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:09:40.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:09:40.031 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:09:40.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:09:40.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:09:40.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:09:41.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:09:41.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:09:41.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:09:41.324 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:09:41.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:09:41.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:09:41.361 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:41.360+0000 7f4ef9357640 -1 osd.0 119 log_to_monitors true 2024-08-19T15:09:41.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:09:42.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:09:42.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:09:42.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:09:42.603 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:09:42.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:09:42.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:09:42.875 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 123 up_thru 123 down_at 120 last_clean_interval [117,119) [v2:127.0.0.1:6802/1291625865,v1:127.0.0.1:6803/1291625865] [v2:127.0.0.1:6804/1291625865,v1:127.0.0.1:6805/1291625865] exists,up 835f901e-853d-4440-b3ef-eda73796c506 2024-08-19T15:09:42.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:09:42.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:09:42.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:09:42.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:09:42.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:09:42.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:09:42.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:09:42.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:09:42.875 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:09:42.875 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:09:42.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:09:42.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:09:42.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:09:42.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:09:42.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:09:42.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:09:42.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:09:42.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:09:42.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:09:43.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:09:43.235 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:09:43.235 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:09:43.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:09:43.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:09:43.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:09:43.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=528280977410 2024-08-19T15:09:43.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 528280977410 2024-08-19T15:09:43.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-528280977410' 2024-08-19T15:09:43.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:09:43.339 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:09:43.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467976 2024-08-19T15:09:43.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467976 2024-08-19T15:09:43.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-528280977410 1-463856467976' 2024-08-19T15:09:43.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:09:43.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:09:43.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318290 2024-08-19T15:09:43.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318290 2024-08-19T15:09:43.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-528280977410 1-463856467976 2-352187318290' 2024-08-19T15:09:43.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:09:43.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-528280977410 2024-08-19T15:09:43.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:09:43.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:09:43.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-528280977410 2024-08-19T15:09:43.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:09:43.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=528280977410 2024-08-19T15:09:43.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 528280977410' 2024-08-19T15:09:43.547 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 528280977410 2024-08-19T15:09:43.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:09:43.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 528280977410 2024-08-19T15:09:43.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:09:44.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:09:44.813 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:09:45.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 528280977410 -lt 528280977410 2024-08-19T15:09:45.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:09:45.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-463856467976 2024-08-19T15:09:45.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:09:45.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:09:45.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-463856467976 2024-08-19T15:09:45.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:09:45.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467976 2024-08-19T15:09:45.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 463856467976' 2024-08-19T15:09:45.091 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 463856467976 2024-08-19T15:09:45.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:09:45.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467976 -lt 463856467976 2024-08-19T15:09:45.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:09:45.398 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318290 2024-08-19T15:09:45.398 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:09:45.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:09:45.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318290 2024-08-19T15:09:45.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:09:45.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318290 2024-08-19T15:09:45.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318290' 2024-08-19T15:09:45.402 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 352187318290 2024-08-19T15:09:45.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:09:45.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318290 -lt 352187318290 2024-08-19T15:09:45.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:09:45.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:09:45.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:09:46.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:09:46.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:09:46.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:09:46.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:09:46.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:09:46.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:09:46.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:09:46.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:09:46.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:09:46.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:09:46.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:09:46.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:09:46.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:09:46.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:09:46.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:09:46.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-08-19T15:09:46.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-08-19T15:09:46.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T15:09:46.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-08-19T15:09:46.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-08-19T15:09:46.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:09:46.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-08-19T15:09:46.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-08-19T15:09:46.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-08-19T15:09:46.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-08-19T15:09:46.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:09:46.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:09:46.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:09:46.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:09:47.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-08-19T15:07:32.247821+0000 2024-08-19T15:09:47.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-08-19T15:09:47.460 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-08-19T15:09:47.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-08-19T15:07:32.247821+0000 2024-08-19T15:09:47.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-08-19T15:09:47.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:07:32.247821+0000 2024-08-19T15:09:47.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:09:47.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:09:47.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:09:47.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-08-19T15:09:47.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:09:47.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:09:47.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:09:47.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:09:47.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:07:32.247821+0000 '>' 2024-08-19T15:07:32.247821+0000 2024-08-19T15:09:47.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:09:48.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:09:48.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:09:48.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-08-19T15:09:48.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:09:48.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:09:48.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:09:48.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:09:49.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:07:32.247821+0000 '>' 2024-08-19T15:07:32.247821+0000 2024-08-19T15:09:49.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:09:50.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:09:50.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:09:50.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-08-19T15:09:50.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:09:50.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:09:50.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:09:50.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:09:50.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:09:48.176100+0000 '>' 2024-08-19T15:07:32.247821+0000 2024-08-19T15:09:50.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:09:50.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-08-19T15:09:50.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-08-19T15:09:50.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-08-19T15:09:50.308 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-08-19T15:09:50.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-08-19T15:09:50.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-08-19T15:09:50.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-08-19T15:09:50.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=123 2024-08-19T15:09:50.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-08-19T15:09:50.342 INFO:tasks.workunit.client.0.smithi102.stderr: . as $in 2024-08-19T15:09:50.342 INFO:tasks.workunit.client.0.smithi102.stderr: | if type == "object" then 2024-08-19T15:09:50.342 INFO:tasks.workunit.client.0.smithi102.stderr: reduce keys[] as $key 2024-08-19T15:09:50.342 INFO:tasks.workunit.client.0.smithi102.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-08-19T15:09:50.342 INFO:tasks.workunit.client.0.smithi102.stderr: elif type == "array" then map( walk(f) ) | f 2024-08-19T15:09:50.342 INFO:tasks.workunit.client.0.smithi102.stderr: else f 2024-08-19T15:09:50.342 INFO:tasks.workunit.client.0.smithi102.stderr: end; 2024-08-19T15:09:50.342 INFO:tasks.workunit.client.0.smithi102.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-08-19T15:09:50.342 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-08-19T15:09:50.342 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-08-19T15:09:50.342 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.version) else . end) 2024-08-19T15:09:50.342 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-08-19T15:09:50.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-08-19T15:09:50.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-08-19T15:09:50.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-08-19T15:09:50.365 INFO:tasks.workunit.client.0.smithi102.stderr: . as $in 2024-08-19T15:09:50.365 INFO:tasks.workunit.client.0.smithi102.stderr: | if type == "object" then 2024-08-19T15:09:50.365 INFO:tasks.workunit.client.0.smithi102.stderr: reduce keys[] as $key 2024-08-19T15:09:50.365 INFO:tasks.workunit.client.0.smithi102.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-08-19T15:09:50.365 INFO:tasks.workunit.client.0.smithi102.stderr: elif type == "array" then map( walk(f) ) | f 2024-08-19T15:09:50.365 INFO:tasks.workunit.client.0.smithi102.stderr: else f 2024-08-19T15:09:50.365 INFO:tasks.workunit.client.0.smithi102.stderr: end; 2024-08-19T15:09:50.365 INFO:tasks.workunit.client.0.smithi102.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-08-19T15:09:50.365 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-08-19T15:09:50.366 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-08-19T15:09:50.366 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.version) else . end) 2024-08-19T15:09:50.366 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-08-19T15:09:50.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-08-19T15:09:50.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-08-19T15:09:50.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-08-19T15:09:50.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-08-19T15:09:50.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-08-19T15:09:50.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-08-19T15:09:50.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-08-19T15:09:50.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-08-19T15:09:50.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:09:50.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:09:50.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:09:50.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:09:50.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:09:50.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-08-19T15:07:32.247821+0000 2024-08-19T15:09:50.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-08-19T15:09:50.900 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-08-19T15:09:50.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-08-19T15:07:32.247821+0000 last_deep_scrub_stamp 2024-08-19T15:09:50.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-08-19T15:09:50.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:07:32.247821+0000 2024-08-19T15:09:50.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-08-19T15:09:50.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:09:50.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:09:50.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:09:50.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:09:50.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:09:50.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:09:50.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:09:51.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:07:32.247821+0000 '>' 2024-08-19T15:07:32.247821+0000 2024-08-19T15:09:51.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:09:52.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:09:52.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:09:52.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:09:52.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:09:52.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:09:52.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:09:52.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:09:52.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:07:32.247821+0000 '>' 2024-08-19T15:07:32.247821+0000 2024-08-19T15:09:52.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:09:53.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:09:53.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:09:53.448 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:09:53.448 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:09:53.449 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:09:53.449 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:09:53.449 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:09:53.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:07:32.247821+0000 '>' 2024-08-19T15:07:32.247821+0000 2024-08-19T15:09:53.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:09:54.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:09:54.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:09:54.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:09:54.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:09:54.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:09:54.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:09:54.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:09:55.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:07:32.247821+0000 '>' 2024-08-19T15:07:32.247821+0000 2024-08-19T15:09:55.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:09:56.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:09:56.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:09:56.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:09:56.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:09:56.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:09:56.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:09:56.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:09:56.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:09:51.312312+0000 '>' 2024-08-19T15:07:32.247821+0000 2024-08-19T15:09:56.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:09:56.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-08-19T15:09:56.308 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-08-19T15:09:56.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-08-19T15:09:56.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-08-19T15:09:56.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-08-19T15:09:56.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-08-19T15:09:56.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-08-19T15:09:56.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=123 2024-08-19T15:09:56.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' false = true ']' 2024-08-19T15:09:56.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq 'def walk(f): 2024-08-19T15:09:56.344 INFO:tasks.workunit.client.0.smithi102.stderr: . as $in 2024-08-19T15:09:56.344 INFO:tasks.workunit.client.0.smithi102.stderr: | if type == "object" then 2024-08-19T15:09:56.344 INFO:tasks.workunit.client.0.smithi102.stderr: reduce keys[] as $key 2024-08-19T15:09:56.344 INFO:tasks.workunit.client.0.smithi102.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-08-19T15:09:56.344 INFO:tasks.workunit.client.0.smithi102.stderr: elif type == "array" then map( walk(f) ) | f 2024-08-19T15:09:56.344 INFO:tasks.workunit.client.0.smithi102.stderr: else f 2024-08-19T15:09:56.344 INFO:tasks.workunit.client.0.smithi102.stderr: end; 2024-08-19T15:09:56.344 INFO:tasks.workunit.client.0.smithi102.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-08-19T15:09:56.344 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-08-19T15:09:56.344 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-08-19T15:09:56.344 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.version) else . end) 2024-08-19T15:09:56.344 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-08-19T15:09:56.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq .inconsistents 2024-08-19T15:09:56.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-08-19T15:09:56.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-08-19T15:09:56.368 INFO:tasks.workunit.client.0.smithi102.stderr: . as $in 2024-08-19T15:09:56.368 INFO:tasks.workunit.client.0.smithi102.stderr: | if type == "object" then 2024-08-19T15:09:56.368 INFO:tasks.workunit.client.0.smithi102.stderr: reduce keys[] as $key 2024-08-19T15:09:56.368 INFO:tasks.workunit.client.0.smithi102.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-08-19T15:09:56.368 INFO:tasks.workunit.client.0.smithi102.stderr: elif type == "array" then map( walk(f) ) | f 2024-08-19T15:09:56.368 INFO:tasks.workunit.client.0.smithi102.stderr: else f 2024-08-19T15:09:56.368 INFO:tasks.workunit.client.0.smithi102.stderr: end; 2024-08-19T15:09:56.368 INFO:tasks.workunit.client.0.smithi102.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-08-19T15:09:56.369 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-08-19T15:09:56.369 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-08-19T15:09:56.369 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.version) else . end) 2024-08-19T15:09:56.369 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-08-19T15:09:56.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-08-19T15:09:56.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-08-19T15:09:56.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-08-19T15:09:56.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-08-19T15:09:56.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-08-19T15:09:56.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-08-19T15:09:56.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-08-19T15:09:56.733 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'ecpool' removed 2024-08-19T15:09:56.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T15:09:56.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:09:56.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:09:56.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:09:56.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:09:56.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:09:56.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:09:56.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:09:56.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:09:56.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:09:56.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:09:56.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:09:56.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:09:56.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:09:56.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:09:56.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:09:56.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:09:56.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:09:56.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:09:56.858 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:09:56.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:09:56.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:09:56.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:09:56.886 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:09:56.887 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:09:56.887 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:09:56.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:09:56.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:09:56.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:09:56.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T15:09:56.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T15:09:56.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T15:09:56.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T15:09:56.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:09:56.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:09:56.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:09:56.889 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:09:56.889 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:09:56.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:09:56.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:09:56.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:09:56.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:09:56.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:09:56.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:09:56.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:09:56.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:09:56.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:09:56.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:09:56.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:09:56.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:09:56.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:09:56.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:09:56.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:09:56.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:09:56.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:09:56.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:09:56.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:09:56.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:09:56.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:09:56.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:09:56.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:09:56.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T15:09:56.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T15:09:56.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:09:56.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:09:56.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T15:09:56.903 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T15:09:56.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T15:09:56.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T15:09:56.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T15:09:56.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2024-08-19T15:09:56.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5750: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2024-08-19T15:09:56.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5751: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2024-08-19T15:09:56.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-08-19T15:09:56.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=true 2024-08-19T15:09:56.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-08-19T15:09:56.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-08-19T15:09:56.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-08-19T15:09:56.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T15:09:56.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T15:09:56.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T15:09:56.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T15:09:56.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T15:09:56.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-08-19T15:09:57.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T15:09:57.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:09:57.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:09:57.274 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:09:57.274 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:09:57.274 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:09:57.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:09:57.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T15:09:57.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T15:09:57.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T15:09:57.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:09:57.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:09:57.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T15:09:57.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T15:09:57.468 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:09:57.468 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:09:57.468 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:09:57.469 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:09:57.469 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:09:57.469 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:09:57.469 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:09:57.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:09:57.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T15:09:57.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T15:09:57.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:09:57.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:09:57.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T15:09:57.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T15:09:57.549 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:09:57.549 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:09:57.549 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:09:57.550 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:09:57.550 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:09:57.550 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:09:57.550 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:09:57.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:09:57.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T15:09:57.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-08-19T15:09:57.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T15:09:57.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T15:09:57.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T15:09:57.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T15:09:57.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T15:09:57.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T15:09:58.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T15:09:58.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:09:58.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:09:58.435 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:09:58.435 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:09:58.435 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:09:58.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:09:58.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:09:58.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:09:58.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-08-19T15:09:58.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-08-19T15:09:58.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-08-19T15:09:58.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:09:58.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:09:58.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T15:09:58.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:09:58.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:09:58.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:09:58.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:09:58.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:09:58.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:09:58.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:09:58.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:09:58.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:09:58.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:09:58.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:09:58.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:09:58.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:09:58.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:09:58.469 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:09:58.469 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:09:58.469 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:09:58.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:09:58.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:09:58.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:09:58.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:09:58.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:09:58.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:09:58.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:09:58.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:09:58.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:09:58.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:09:58.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:09:58.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:09:58.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:09:58.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:09:58.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:09:58.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=791bfa96-95df-4f85-9b52-0573c1cfef78 2024-08-19T15:09:58.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 791bfa96-95df-4f85-9b52-0573c1cfef78' 2024-08-19T15:09:58.472 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 791bfa96-95df-4f85-9b52-0573c1cfef78 2024-08-19T15:09:58.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:09:58.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGYMNmKPT+HBAAMVmV4M32hDhIT8zIUnIoVQ== 2024-08-19T15:09:58.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGYMNmKPT+HBAAMVmV4M32hDhIT8zIUnIoVQ=="}' 2024-08-19T15:09:58.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 791bfa96-95df-4f85-9b52-0573c1cfef78 -i td/osd-scrub-repair/0/new.json 2024-08-19T15:09:58.691 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:09:58.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T15:09:58.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGYMNmKPT+HBAAMVmV4M32hDhIT8zIUnIoVQ== --osd-uuid 791bfa96-95df-4f85-9b52-0573c1cfef78 2024-08-19T15:09:58.730 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:58.729+0000 7f040700d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:09:58.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:58.741+0000 7f040700d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:09:58.743 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:58.742+0000 7f040700d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:09:58.744 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:58.743+0000 7f040700d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:09:58.744 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:58.743+0000 7f040700d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:09:58.744 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:58.743+0000 7f040700d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:09:58.833 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:09:58.832+0000 7f040700d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T15:10:04.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T15:10:04.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:10:04.249 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T15:10:04.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T15:10:04.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:10:04.597 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:10:04.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T15:10:04.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:10:04.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:10:04.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:10:04.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:10:04.623 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:10:04.622+0000 7fc4b0933640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:10:04.625 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:10:04.624+0000 7fc4b0933640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:10:04.626 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:10:04.625+0000 7fc4b0933640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:10:04.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T15:10:04.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:10:04.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:10:04.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:10:04.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:10:04.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:04.864 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:10:04.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:10:04.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:04.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:10:05.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:05.147 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:10:05.146+0000 7fc4b0933640 -1 Falling back to public interface 2024-08-19T15:10:06.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:06.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:06.131 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:10:06.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:10:06.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:06.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:10:06.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:07.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:07.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:07.403 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:10:07.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:10:07.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:07.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:10:07.441 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:10:07.440+0000 7fc4b0933640 -1 osd.0 0 log_to_monitors true 2024-08-19T15:10:07.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:08.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:08.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:08.700 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:10:08.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:10:08.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:08.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:10:08.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:09.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:09.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:09.988 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:10:09.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:10:09.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:09.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:10:10.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:11.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:11.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:11.340 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:10:11.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:10:11.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:11.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:10:11.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:12.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:12.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:12.616 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:10:12.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:10:12.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:12.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:10:12.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:13.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:13.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:13.894 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:10:13.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:10:13.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:13.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:10:14.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:15.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:15.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:15.173 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:10:15.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:10:15.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:15.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:10:15.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:16.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:16.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:16.456 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:10:16.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:10:16.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:16.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:10:16.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:17.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:17.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:17.731 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:10:17.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:10:17.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:17.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:10:18.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:19.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:19.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:19.011 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:10:19.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:10:19.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:19.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:10:19.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:20.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:20.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:20.502 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:10:20.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:10:20.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:20.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:10:21.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:22.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:22.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:22.056 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:10:22.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:10:22.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:22.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:10:22.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:23.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:23.345 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:10:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:10:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:10:24.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:25.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:25.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:25.053 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:10:25.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:10:25.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:25.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:10:25.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:26.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:26.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:26.797 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:10:26.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:10:26.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:26.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:10:27.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:28.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:28.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:28.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:10:28.305 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:10:28.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:28.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:10:28.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:29.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:29.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:29.655 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:10:29.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:10:29.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:29.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:10:29.937 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2500282097,v1:127.0.0.1:6803/2500282097] [v2:127.0.0.1:6804/2500282097,v1:127.0.0.1:6805/2500282097] exists,up 791bfa96-95df-4f85-9b52-0573c1cfef78 2024-08-19T15:10:29.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:10:29.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:10:29.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:10:29.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-08-19T15:10:29.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-08-19T15:10:29.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:10:29.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:10:29.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T15:10:29.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:10:29.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:10:29.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:10:29.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:10:29.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:10:29.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:10:29.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:10:29.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:10:29.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:10:29.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:10:29.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:10:29.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:10:29.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:10:29.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:10:29.943 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:10:29.943 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:10:29.943 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:10:29.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:10:29.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:10:29.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:10:29.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:10:29.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:10:29.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:10:29.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:10:29.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:10:29.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:10:29.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:10:29.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:10:29.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:10:29.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:10:29.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:10:29.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:10:29.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c78dc1f0-99d1-4c82-8258-f34876d36832 2024-08-19T15:10:29.945 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 c78dc1f0-99d1-4c82-8258-f34876d36832 2024-08-19T15:10:29.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c78dc1f0-99d1-4c82-8258-f34876d36832' 2024-08-19T15:10:29.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:10:29.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBlYMNmn6YhORAAKItnoDwPlzDopb6oPaO59A== 2024-08-19T15:10:29.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBlYMNmn6YhORAAKItnoDwPlzDopb6oPaO59A=="}' 2024-08-19T15:10:29.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c78dc1f0-99d1-4c82-8258-f34876d36832 -i td/osd-scrub-repair/1/new.json 2024-08-19T15:10:30.370 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:10:30.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T15:10:30.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBlYMNmn6YhORAAKItnoDwPlzDopb6oPaO59A== --osd-uuid c78dc1f0-99d1-4c82-8258-f34876d36832 2024-08-19T15:10:30.411 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:10:30.410+0000 7fd99e72c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:10:30.447 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:10:30.446+0000 7fd99e72c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:10:30.447 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:10:30.447+0000 7fd99e72c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:10:30.448 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:10:30.447+0000 7fd99e72c640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:10:30.448 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:10:30.447+0000 7fd99e72c640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:10:30.448 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:10:30.447+0000 7fd99e72c640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:10:30.521 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:10:30.520+0000 7fd99e72c640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T15:10:36.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T15:10:36.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:10:36.260 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T15:10:36.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T15:10:36.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:10:36.641 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:10:36.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T15:10:36.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:10:36.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:10:36.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:10:36.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:10:36.668 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:10:36.667+0000 7fee9a80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:10:36.669 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:10:36.669+0000 7fee9a80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:10:36.670 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:10:36.670+0000 7fee9a80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:10:36.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T15:10:36.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:10:36.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:10:36.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:10:36.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:10:36.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:36.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:10:36.909 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:10:36.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:36.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:10:37.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:37.198 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:10:37.198+0000 7fee9a80f640 -1 Falling back to public interface 2024-08-19T15:10:38.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:38.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:38.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:10:38.175 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:10:38.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:38.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:10:38.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:39.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:39.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:39.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:10:39.454 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:10:39.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:39.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:10:39.522 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:10:39.521+0000 7fee9a80f640 -1 osd.1 0 log_to_monitors true 2024-08-19T15:10:39.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:40.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:40.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:40.730 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:10:40.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:10:40.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:40.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:10:41.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:42.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:42.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:42.121 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:10:42.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:10:42.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:42.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:10:42.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:43.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:43.492 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:10:43.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:43.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:10:43.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:43.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:10:43.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:44.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:44.768 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:10:44.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:44.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:10:44.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:44.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:10:45.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:46.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:46.093 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:10:46.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:46.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:10:46.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:46.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:10:46.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:47.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:47.478 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:10:47.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:47.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:10:47.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:47.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:10:47.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:48.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:48.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:48.756 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:10:48.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:10:48.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:48.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:10:49.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:50.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:50.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:50.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:10:50.103 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:10:50.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:50.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:10:50.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:51.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:51.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:51.492 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:10:51.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:10:51.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:51.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:10:51.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:52.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:52.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:52.765 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:10:52.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:10:52.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:52.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:10:53.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:54.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:54.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:54.483 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:10:54.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:10:54.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:54.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:10:55.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:56.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:56.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:56.091 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:10:56.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:10:56.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:56.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:10:56.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:57.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:57.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:57.514 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:10:57.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:10:57.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:57.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:10:58.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:10:59.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:10:59.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:10:59.153 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:10:59.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:10:59.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:10:59.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:10:59.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:00.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:00.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:00.426 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:11:00.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:11:00.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:00.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:11:00.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:01.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:01.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:01.761 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:11:01.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:11:01.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:01.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:11:02.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:03.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:03.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:03.066 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T15:11:03.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T15:11:03.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:03.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:11:03.337 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3309885350,v1:127.0.0.1:6811/3309885350] [v2:127.0.0.1:6812/3309885350,v1:127.0.0.1:6813/3309885350] exists,up c78dc1f0-99d1-4c82-8258-f34876d36832 2024-08-19T15:11:03.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:11:03.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:11:03.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:11:03.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-08-19T15:11:03.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-08-19T15:11:03.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:11:03.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:11:03.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T15:11:03.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:11:03.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:11:03.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:11:03.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:11:03.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:11:03.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:11:03.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T15:11:03.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T15:11:03.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:11:03.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:11:03.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:11:03.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:11:03.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:11:03.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:11:03.343 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:11:03.343 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:11:03.343 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:11:03.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:11:03.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:11:03.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:11:03.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:11:03.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:11:03.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:11:03.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:11:03.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:11:03.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:11:03.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:11:03.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:11:03.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:11:03.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:11:03.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T15:11:03.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:11:03.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8d5c26d9-bfaa-498e-b826-ea3d9adf813b 2024-08-19T15:11:03.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 8d5c26d9-bfaa-498e-b826-ea3d9adf813b' 2024-08-19T15:11:03.345 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 8d5c26d9-bfaa-498e-b826-ea3d9adf813b 2024-08-19T15:11:03.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:11:03.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCHYMNmzZdgFRAADQu7Eu38BQqMmzE79z510w== 2024-08-19T15:11:03.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCHYMNmzZdgFRAADQu7Eu38BQqMmzE79z510w=="}' 2024-08-19T15:11:03.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8d5c26d9-bfaa-498e-b826-ea3d9adf813b -i td/osd-scrub-repair/2/new.json 2024-08-19T15:11:03.690 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:11:03.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-08-19T15:11:03.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHYMNmzZdgFRAADQu7Eu38BQqMmzE79z510w== --osd-uuid 8d5c26d9-bfaa-498e-b826-ea3d9adf813b 2024-08-19T15:11:03.731 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:11:03.730+0000 7f34790f0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:11:03.768 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:11:03.767+0000 7f34790f0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:11:03.769 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:11:03.768+0000 7f34790f0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:11:03.770 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:11:03.769+0000 7f34790f0640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:11:03.770 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:11:03.769+0000 7f34790f0640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:11:03.770 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:11:03.769+0000 7f34790f0640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:11:03.843 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:11:03.842+0000 7f34790f0640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-08-19T15:11:09.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-08-19T15:11:09.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:11:09.404 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T15:11:09.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T15:11:09.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:11:09.751 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T15:11:09.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T15:11:09.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:11:09.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:11:09.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:11:09.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:11:09.778 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:11:09.777+0000 7f86c08d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:11:09.780 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:11:09.779+0000 7f86c08d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:11:09.781 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:11:09.780+0000 7f86c08d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:11:10.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T15:11:10.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:11:10.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T15:11:10.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:11:10.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:11:10.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:10.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:11:10.019 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:11:10.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:10.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:11:10.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:10.305 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:11:10.304+0000 7f86c08d7640 -1 Falling back to public interface 2024-08-19T15:11:11.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:11.287 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:11:11.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:11.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:11:11.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:11.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:11:11.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:12.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:12.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:12.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:11:12.560 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:11:12.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:12.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:11:12.741 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:11:12.739+0000 7f86c08d7640 -1 osd.2 0 log_to_monitors true 2024-08-19T15:11:12.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:13.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:13.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:13.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:11:13.934 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:11:13.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:13.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:11:14.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:15.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:15.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:15.209 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:11:15.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:11:15.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:15.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:11:15.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:16.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:16.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:16.582 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:11:16.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:11:16.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:16.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:11:17.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:18.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:18.016 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:11:18.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:18.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:11:18.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:18.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:11:18.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:19.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:19.294 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:11:19.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:19.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:11:19.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:19.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:11:19.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:20.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:20.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:20.568 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:11:20.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:11:20.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:20.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:11:20.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:21.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:21.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:21.972 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:11:21.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:11:21.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:21.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:11:22.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:23.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:23.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:23.251 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:11:23.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:11:23.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:23.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:11:23.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:24.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:24.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:24.549 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:11:24.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:11:24.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:24.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:11:25.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:26.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:26.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:26.224 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:11:26.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:11:26.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:26.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:11:26.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:27.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:27.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:27.803 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:11:27.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:11:27.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:27.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:11:28.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:29.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:29.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:29.150 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:11:29.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:11:29.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:29.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:11:29.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:30.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:30.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:30.716 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:11:30.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:11:30.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:30.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:11:31.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:32.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:32.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:32.326 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:11:32.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:11:32.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:32.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:11:32.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:33.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:33.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:33.838 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:11:33.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:11:33.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:33.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:11:34.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:35.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:35.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:35.599 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:11:35.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:11:35.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:35.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:11:35.882 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3418247091,v1:127.0.0.1:6819/3418247091] [v2:127.0.0.1:6820/3418247091,v1:127.0.0.1:6821/3418247091] exists,up 8d5c26d9-bfaa-498e-b826-ea3d9adf813b 2024-08-19T15:11:35.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:11:35.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:11:35.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:11:35.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-08-19T15:11:35.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T15:11:36.145 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-08-19T15:11:36.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T15:11:36.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T15:11:36.528 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' created 2024-08-19T15:11:36.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:11:37.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T15:11:39.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-08-19T15:11:39.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-08-19T15:11:39.798 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'foo' created 2024-08-19T15:11:39.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:11:40.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2024-08-19T15:11:40.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-08-19T15:11:40.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-08-19T15:11:40.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-08-19T15:11:40.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-08-19T15:11:40.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-08-19T15:11:41.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-08-19T15:11:41.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-08-19T15:11:42.151 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'ecpool' created 2024-08-19T15:11:42.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:11:43.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-08-19T15:11:43.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-08-19T15:11:43.776 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 3 allow_ec_overwrites to true 2024-08-19T15:11:43.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-08-19T15:11:43.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:11:43.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:11:43.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:11:43.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:11:43.790 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:11:43.790 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:11:43.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:11:43.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:11:43.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:11:43.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:11:43.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:11:43.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:11:43.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:11:43.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:11:43.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:11:44.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:11:44.148 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:11:44.148 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:11:44.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:11:44.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:11:44.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:11:44.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-08-19T15:11:44.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-08-19T15:11:44.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-08-19T15:11:44.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:11:44.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:11:44.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-08-19T15:11:44.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-08-19T15:11:44.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672974' 2024-08-19T15:11:44.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:11:44.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:11:44.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-08-19T15:11:44.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-08-19T15:11:44.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672974 2-64424509447' 2024-08-19T15:11:44.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:11:44.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-08-19T15:11:44.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:11:44.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:11:44.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-08-19T15:11:44.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:11:44.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-08-19T15:11:44.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-08-19T15:11:44.478 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836500 2024-08-19T15:11:44.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:11:44.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-08-19T15:11:44.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:11:44.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-08-19T15:11:44.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:11:44.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:11:44.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-08-19T15:11:44.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:11:44.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-08-19T15:11:44.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-08-19T15:11:44.761 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672974 2024-08-19T15:11:44.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:11:45.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-08-19T15:11:45.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:11:45.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-08-19T15:11:45.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:11:45.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:11:45.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-08-19T15:11:45.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:11:45.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-08-19T15:11:45.030 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509447 2024-08-19T15:11:45.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-08-19T15:11:45.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:11:45.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509447 2024-08-19T15:11:45.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:11:46.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:11:46.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:11:46.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509447 2024-08-19T15:11:46.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:11:46.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:11:46.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:11:46.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:11:46.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:11:46.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:11:46.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:11:46.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:11:46.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:11:46.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:11:46.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:11:47.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:11:47.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:11:47.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:11:47.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:11:47.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:11:47.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:11:47.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:11:47.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-08-19T15:11:47.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-08-19T15:11:47.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:11:47.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:11:47.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:11:47.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:11:47.531 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:11:47.531 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:11:47.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:11:47.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:11:47.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:11:47.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:11:47.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:11:47.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:11:47.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:11:47.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:11:47.623 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:11:47.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:11:47.886 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:11:47.886 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:11:47.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:11:47.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:11:47.887 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:11:47.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-08-19T15:11:47.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-08-19T15:11:47.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-08-19T15:11:47.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:11:47.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:11:48.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-08-19T15:11:48.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-08-19T15:11:48.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672976' 2024-08-19T15:11:48.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:11:48.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:11:48.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-08-19T15:11:48.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-08-19T15:11:48.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672976 2-64424509449' 2024-08-19T15:11:48.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:11:48.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-08-19T15:11:48.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:11:48.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:11:48.301 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-08-19T15:11:48.301 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:11:48.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-08-19T15:11:48.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-08-19T15:11:48.302 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836502 2024-08-19T15:11:48.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:11:48.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-08-19T15:11:48.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:11:48.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-08-19T15:11:48.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:11:48.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:11:48.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-08-19T15:11:48.573 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:11:48.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-08-19T15:11:48.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-08-19T15:11:48.575 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672976 2024-08-19T15:11:48.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:11:48.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-08-19T15:11:48.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:11:48.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-08-19T15:11:48.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:11:48.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:11:48.860 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-08-19T15:11:48.860 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:11:48.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-08-19T15:11:48.861 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509449 2024-08-19T15:11:48.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-08-19T15:11:48.861 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:11:49.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-08-19T15:11:49.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:11:49.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:11:49.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:11:49.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:11:49.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:11:49.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:11:49.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:11:49.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:11:49.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:11:49.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:11:49.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:11:49.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:11:49.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:11:49.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:11:49.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:11:50.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:11:50.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:11:50.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:11:50.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-08-19T15:11:50.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-08-19T15:11:50.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-08-19T15:11:50.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-08-19T15:11:50.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:11:50.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:11:50.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-08-19T15:11:50.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:11:50.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:11:50.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:11:50.413 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:11:50.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:11:50.781 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:11:50.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:11:50.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:11:50.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:11:51.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-08-19T15:11:51.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-08-19T15:11:51.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-08-19T15:11:51.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-08-19T15:11:51.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-08-19T15:11:51.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:11:51.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:11:51.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:11:51.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:11:51.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:11:51.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:11:51.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:11:51.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:11:51.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:11:51.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:11:51.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:11:51.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:11:51.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:11:51.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:11:51.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:11:51.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:11:51.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:11:51.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:11:51.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:11:51.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:11:51.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:11:51.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:11:51.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:11:51.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:11:53.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:11:53.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:11:53.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:11:53.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:11:53.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:11:53.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:11:53.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:11:53.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:11:53.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:11:53.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:11:53.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:11:53.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:11:53.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:11:53.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:11:53.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:11:53.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:11:53.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:11:53.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:11:53.759 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:11:53.759 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:11:53.760 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:11:53.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:11:53.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:11:53.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:11:53.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:11:53.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:11:53.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:11:53.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:11:53.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:11:53.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:11:53.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:11:53.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:11:53.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:11:53.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:11:53.761 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:11:53.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:11:53.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:11:53.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:11:53.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:11:53.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:11:53.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:11:53.790 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:11:53.789+0000 7f463e21c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:11:53.792 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:11:53.791+0000 7f463e21c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:11:53.794 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:11:53.793+0000 7f463e21c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:11:54.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:11:54.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:11:54.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:11:54.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:11:54.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:11:54.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:54.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:11:54.039 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:11:54.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:54.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:11:54.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:54.316 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:11:54.315+0000 7f463e21c640 -1 Falling back to public interface 2024-08-19T15:11:55.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:55.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:55.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:11:55.305 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:11:55.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:55.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:11:55.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:56.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:56.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:56.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:11:56.577 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:11:56.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:56.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:11:56.805 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:11:56.804+0000 7f463e21c640 -1 osd.1 34 log_to_monitors true 2024-08-19T15:11:56.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:11:57.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:11:57.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:11:57.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:11:57.909 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:11:57.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:11:57.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:11:58.200 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 39 up_thru 39 down_at 35 last_clean_interval [10,34) [v2:127.0.0.1:6810/2132246113,v1:127.0.0.1:6811/2132246113] [v2:127.0.0.1:6812/2132246113,v1:127.0.0.1:6813/2132246113] exists,up c78dc1f0-99d1-4c82-8258-f34876d36832 2024-08-19T15:11:58.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:11:58.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:11:58.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:11:58.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:11:58.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:11:58.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:11:58.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:11:58.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:11:58.201 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:11:58.201 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:11:58.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:11:58.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:11:58.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:11:58.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:11:58.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:11:58.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:11:58.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:11:58.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:11:58.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:11:58.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:11:58.712 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:11:58.712 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:11:58.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:11:58.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:11:58.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:11:58.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-08-19T15:11:58.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-08-19T15:11:58.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-08-19T15:11:58.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:11:58.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:11:58.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724546 2024-08-19T15:11:58.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724546 2024-08-19T15:11:58.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-167503724546' 2024-08-19T15:11:58.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:11:58.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:11:59.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-08-19T15:11:59.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-08-19T15:11:59.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-167503724546 2-64424509452' 2024-08-19T15:11:59.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:11:59.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-08-19T15:11:59.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:11:59.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:11:59.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-08-19T15:11:59.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:11:59.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-08-19T15:11:59.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-08-19T15:11:59.063 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836505 2024-08-19T15:11:59.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:11:59.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-08-19T15:11:59.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:12:00.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:12:00.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:12:00.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836505 2024-08-19T15:12:00.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:12:00.605 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724546 2024-08-19T15:12:00.606 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:12:00.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:12:00.607 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724546 2024-08-19T15:12:00.607 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:12:00.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724546 2024-08-19T15:12:00.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724546' 2024-08-19T15:12:00.609 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 167503724546 2024-08-19T15:12:00.609 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:12:00.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724546 -lt 167503724546 2024-08-19T15:12:00.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:12:00.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-08-19T15:12:00.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:12:00.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:12:00.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-08-19T15:12:00.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:12:00.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-08-19T15:12:00.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-08-19T15:12:00.896 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509452 2024-08-19T15:12:00.896 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:12:01.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-08-19T15:12:01.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:12:01.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:12:01.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:12:01.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:12:01.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:12:01.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:12:01.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:12:01.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:12:01.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:12:01.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:12:01.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:12:01.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:12:01.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:12:01.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:12:01.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:12:02.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:12:02.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:12:02.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:12:02.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-08-19T15:12:02.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-08-19T15:12:02.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-08-19T15:12:02.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:12:02.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:12:02.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-08-19T15:12:02.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:12:02.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:12:02.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:12:02.514 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:12:02.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:12:02.891 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:12:02.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:12:02.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:12:02.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:12:03.396 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-08-19T15:12:03.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-08-19T15:12:03.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-08-19T15:12:03.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-08-19T15:12:03.399 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records in 2024-08-19T15:12:03.399 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records out 2024-08-19T15:12:03.399 INFO:tasks.workunit.client.0.smithi102.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 6.5809e-05 s, 31.1 MB/s 2024-08-19T15:12:03.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:12:03.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:12:03.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:12:03.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:12:03.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:12:03.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:12:03.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:12:03.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:12:03.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:12:03.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:12:03.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:12:03.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:12:03.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:12:03.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:12:03.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:12:03.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:12:03.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:12:03.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:12:03.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:12:03.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:12:03.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:12:03.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:12:03.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:12:03.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:12:05.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:12:05.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:12:05.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:12:05.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:12:05.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:12:05.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:12:05.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:12:05.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:12:05.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:12:05.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:12:05.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:12:05.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:12:05.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:12:05.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:12:05.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:12:05.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:12:05.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:12:05.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:12:05.987 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:12:05.987 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:12:05.987 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:12:05.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:12:05.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:12:05.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:12:05.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:12:05.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:12:05.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:12:05.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:12:05.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:12:05.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:12:05.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:12:05.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:12:05.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:12:05.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:12:05.988 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:12:05.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:12:05.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:12:05.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:12:05.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:12:05.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:12:05.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:12:06.014 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:06.013+0000 7fd51ca1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:12:06.016 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:06.015+0000 7fd51ca1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:12:06.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:06.016+0000 7fd51ca1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:12:06.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:12:06.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:12:06.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:12:06.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:12:06.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:12:06.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:06.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:12:06.263 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:12:06.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:06.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:12:06.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:12:06.541 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:06.540+0000 7fd51ca1e640 -1 Falling back to public interface 2024-08-19T15:12:07.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:12:07.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:07.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:12:07.532 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:12:07.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:07.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:12:07.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:12:08.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:12:08.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:08.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:12:08.804 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:12:08.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:08.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:12:09.055 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:09.054+0000 7fd51ca1e640 -1 osd.0 43 log_to_monitors true 2024-08-19T15:12:09.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:12:10.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:12:10.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:10.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:12:10.085 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:12:10.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:10.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:12:10.408 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 48 up_thru 48 down_at 44 last_clean_interval [5,43) [v2:127.0.0.1:6802/4101949252,v1:127.0.0.1:6803/4101949252] [v2:127.0.0.1:6804/4101949252,v1:127.0.0.1:6805/4101949252] exists,up 791bfa96-95df-4f85-9b52-0573c1cfef78 2024-08-19T15:12:10.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:12:10.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:12:10.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:12:10.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:12:10.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:12:10.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:12:10.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:12:10.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:12:10.409 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:12:10.409 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:12:10.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:12:10.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:12:10.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:12:10.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:12:10.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:12:10.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:12:10.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:12:10.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:12:10.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:12:10.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:12:10.912 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:12:10.913 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:12:10.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:12:10.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:12:10.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:12:11.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430210 2024-08-19T15:12:11.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430210 2024-08-19T15:12:11.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210' 2024-08-19T15:12:11.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:12:11.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:12:11.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724549 2024-08-19T15:12:11.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724549 2024-08-19T15:12:11.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-167503724549' 2024-08-19T15:12:11.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:12:11.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:12:11.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-08-19T15:12:11.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-08-19T15:12:11.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-167503724549 2-64424509455' 2024-08-19T15:12:11.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:12:11.222 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430210 2024-08-19T15:12:11.222 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:12:11.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:12:11.223 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430210 2024-08-19T15:12:11.224 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:12:11.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430210 2024-08-19T15:12:11.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430210' 2024-08-19T15:12:11.225 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 206158430210 2024-08-19T15:12:11.225 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:12:11.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 206158430210 2024-08-19T15:12:11.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:12:12.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:12:12.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:12:12.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430210 -lt 206158430210 2024-08-19T15:12:12.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:12:12.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724549 2024-08-19T15:12:12.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:12:12.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:12:12.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724549 2024-08-19T15:12:12.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:12:12.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724549 2024-08-19T15:12:12.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724549' 2024-08-19T15:12:12.765 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 167503724549 2024-08-19T15:12:12.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:12:13.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724550 -lt 167503724549 2024-08-19T15:12:13.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:12:13.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-08-19T15:12:13.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:12:13.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:12:13.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-08-19T15:12:13.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:12:13.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-08-19T15:12:13.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-08-19T15:12:13.066 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509455 2024-08-19T15:12:13.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:12:13.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509455 2024-08-19T15:12:13.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:12:13.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:12:13.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:12:13.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:12:13.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:12:13.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:12:13.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:12:13.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:12:13.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:12:13.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:12:13.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:12:13.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:12:13.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:12:13.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:12:13.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:12:14.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:12:14.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:12:14.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:12:14.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-08-19T15:12:14.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-08-19T15:12:14.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-08-19T15:12:14.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:12:14.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:12:14.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-08-19T15:12:14.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:12:14.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:12:14.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:12:14.673 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:12:14.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:12:15.041 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:12:15.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:12:15.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:12:15.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:12:15.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-08-19T15:12:15.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-08-19T15:12:15.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-08-19T15:12:15.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-08-19T15:12:15.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:12:15.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:12:15.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:12:15.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:12:15.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-08-19T15:12:15.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:12:15.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:12:15.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:12:15.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:12:15.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:12:15.333 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:12:15.333 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:12:15.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:12:15.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:12:15.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:12:15.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:12:15.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-08-19T15:12:15.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:12:15.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:12:15.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:12:15.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:12:15.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:12:15.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-08-19T15:12:17.231 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-08-19T15:12:17.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:12:17.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:12:17.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:12:17.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:12:17.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:12:17.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:12:17.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:12:17.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:12:17.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:12:17.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:12:17.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:12:17.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:12:17.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:12:17.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:12:17.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:12:17.750 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:12:17.750 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:12:17.750 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:12:17.750 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:12:17.750 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:12:17.750 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:12:17.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:12:17.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:12:17.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:12:17.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:12:17.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:12:17.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:12:17.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:12:17.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:12:17.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:12:17.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:12:17.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:12:17.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:12:17.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:12:17.753 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:12:17.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:12:17.753 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:12:17.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:12:17.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:12:17.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:12:17.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:12:17.780 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:17.779+0000 7fe837a30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:12:17.782 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:17.781+0000 7fe837a30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:12:17.783 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:17.782+0000 7fe837a30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:12:18.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:12:18.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:12:18.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:12:18.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:12:18.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:12:18.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:18.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:12:18.032 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:12:18.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:18.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:12:18.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:12:18.308 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:18.308+0000 7fe837a30640 -1 Falling back to public interface 2024-08-19T15:12:19.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:12:19.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:19.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:12:19.311 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:12:19.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:19.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:12:19.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:12:20.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:12:20.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:20.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:12:20.570 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:12:20.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:20.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:12:20.679 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:20.678+0000 7fe837a30640 -1 osd.1 52 log_to_monitors true 2024-08-19T15:12:20.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:12:21.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:12:21.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:21.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:12:21.910 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:12:21.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:21.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:12:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 53 last_clean_interval [39,52) [v2:127.0.0.1:6810/99802252,v1:127.0.0.1:6811/99802252] [v2:127.0.0.1:6812/99802252,v1:127.0.0.1:6813/99802252] exists,up c78dc1f0-99d1-4c82-8258-f34876d36832 2024-08-19T15:12:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:12:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:12:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:12:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:12:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:12:22.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:12:22.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:12:22.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:12:22.302 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:12:22.302 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:12:22.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:12:22.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:12:22.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:12:22.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:12:22.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:12:22.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:12:22.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:12:22.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:12:22.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:12:22.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:12:22.758 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:12:22.758 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:12:22.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:12:22.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:12:22.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:12:22.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430213 2024-08-19T15:12:22.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430213 2024-08-19T15:12:22.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430213' 2024-08-19T15:12:22.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:12:22.903 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:12:23.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135874 2024-08-19T15:12:23.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135874 2024-08-19T15:12:23.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430213 1-244813135874' 2024-08-19T15:12:23.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:12:23.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:12:23.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-08-19T15:12:23.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-08-19T15:12:23.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430213 1-244813135874 2-64424509459' 2024-08-19T15:12:23.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:12:23.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430213 2024-08-19T15:12:23.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:12:23.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:12:23.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430213 2024-08-19T15:12:23.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:12:23.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430213 2024-08-19T15:12:23.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430213' 2024-08-19T15:12:23.112 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 206158430213 2024-08-19T15:12:23.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:12:23.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430212 -lt 206158430213 2024-08-19T15:12:23.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:12:24.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:12:24.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:12:24.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430214 -lt 206158430213 2024-08-19T15:12:24.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:12:24.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135874 2024-08-19T15:12:24.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:12:24.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:12:24.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135874 2024-08-19T15:12:24.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:12:24.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135874 2024-08-19T15:12:24.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-08-19T15:12:24.663 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 244813135874 2024-08-19T15:12:24.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:12:24.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135874 -lt 244813135874 2024-08-19T15:12:24.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:12:24.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-08-19T15:12:24.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:12:24.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:12:24.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-08-19T15:12:24.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:12:24.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-08-19T15:12:24.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-08-19T15:12:24.979 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509459 2024-08-19T15:12:24.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:12:25.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-08-19T15:12:25.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:12:25.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:12:25.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:12:25.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:12:25.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:12:25.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:12:25.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:12:25.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:12:25.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:12:25.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:12:25.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:12:25.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:12:25.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:12:25.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:12:25.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:12:26.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:12:26.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:12:26.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:12:26.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-08-19T15:12:26.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-08-19T15:12:26.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-08-19T15:12:26.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:12:26.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:12:26.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-08-19T15:12:26.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:12:26.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:12:26.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:12:26.547 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:12:26.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:12:27.041 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:12:27.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:12:27.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:12:27.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:12:27.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-08-19T15:12:27.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-08-19T15:12:27.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-08-19T15:12:27.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-08-19T15:12:27.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-08-19T15:12:27.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-08-19T15:12:27.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-08-19T15:12:27.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:12:27.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:12:27.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:12:27.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:12:27.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-08-19T15:12:27.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:12:27.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:12:27.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:12:27.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:12:27.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:12:27.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:12:27.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:12:27.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:12:27.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:12:27.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:12:27.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:12:27.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-08-19T15:12:27.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:12:27.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:12:27.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:12:27.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:12:27.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:12:27.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-08-19T15:12:29.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:12:29.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:12:29.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:12:29.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:12:29.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:12:29.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:12:29.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:12:29.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:12:29.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:12:29.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:12:29.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:12:29.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:12:29.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:12:29.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:12:29.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:12:29.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:12:29.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:12:29.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:12:29.969 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:12:29.969 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:12:29.969 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:12:29.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:12:29.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:12:29.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:12:29.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:12:29.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:12:29.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:12:29.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:12:29.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:12:29.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:12:29.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:12:29.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:12:29.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:12:29.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:12:29.971 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:12:29.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:12:29.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:12:29.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:12:29.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:12:29.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:12:29.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:12:29.998 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:29.996+0000 7f07ab211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:12:29.999 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:29.998+0000 7f07ab211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:12:30.001 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:30.000+0000 7f07ab211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:12:30.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:12:30.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:12:30.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:12:30.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:12:30.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:12:30.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:30.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:12:30.245 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:12:30.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:30.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:12:30.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:12:30.517 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:30.517+0000 7f07ab211640 -1 Falling back to public interface 2024-08-19T15:12:31.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:12:31.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:31.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:12:31.512 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:12:31.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:31.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:12:31.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:12:32.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:12:32.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:32.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:12:32.783 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:12:32.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:32.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:12:32.931 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:32.930+0000 7f07ab211640 -1 osd.0 61 log_to_monitors true 2024-08-19T15:12:33.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:12:34.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:12:34.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:34.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:12:34.063 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:12:34.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:34.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:12:34.703 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 66 up_thru 66 down_at 62 last_clean_interval [48,61) [v2:127.0.0.1:6802/3799610177,v1:127.0.0.1:6803/3799610177] [v2:127.0.0.1:6804/3799610177,v1:127.0.0.1:6805/3799610177] exists,up 791bfa96-95df-4f85-9b52-0573c1cfef78 2024-08-19T15:12:34.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:12:34.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:12:34.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:12:34.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:12:34.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:12:34.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:12:34.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:12:34.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:12:34.704 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:12:34.704 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:12:34.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:12:34.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:12:34.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:12:34.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:12:34.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:12:34.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:12:34.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:12:34.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:12:34.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:12:35.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:12:35.063 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:12:35.063 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:12:35.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:12:35.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:12:35.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:12:35.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-08-19T15:12:35.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-08-19T15:12:35.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538' 2024-08-19T15:12:35.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:12:35.179 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:12:35.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135877 2024-08-19T15:12:35.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135877 2024-08-19T15:12:35.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-244813135877' 2024-08-19T15:12:35.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:12:35.282 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:12:35.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-08-19T15:12:35.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-08-19T15:12:35.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-244813135877 2-64424509462' 2024-08-19T15:12:35.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:12:35.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841538 2024-08-19T15:12:35.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:12:35.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:12:35.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841538 2024-08-19T15:12:35.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:12:35.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-08-19T15:12:35.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841538' 2024-08-19T15:12:35.387 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 283467841538 2024-08-19T15:12:35.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:12:35.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 283467841538 2024-08-19T15:12:35.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:12:36.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:12:36.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:12:36.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841538 -lt 283467841538 2024-08-19T15:12:36.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:12:36.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135877 2024-08-19T15:12:36.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:12:36.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:12:36.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135877 2024-08-19T15:12:36.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:12:36.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135877 2024-08-19T15:12:36.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135877' 2024-08-19T15:12:36.935 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 244813135877 2024-08-19T15:12:36.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:12:37.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135878 -lt 244813135877 2024-08-19T15:12:37.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:12:37.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-08-19T15:12:37.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:12:37.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:12:37.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-08-19T15:12:37.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:12:37.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-08-19T15:12:37.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-08-19T15:12:37.255 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 64424509462 2024-08-19T15:12:37.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:12:37.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509462 2024-08-19T15:12:37.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:12:37.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:12:37.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:12:37.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:12:37.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:12:37.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:12:37.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:12:37.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:12:37.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:12:37.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:12:37.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:12:38.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:12:38.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:12:38.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:12:38.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:12:38.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:12:38.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:12:38.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:12:38.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-08-19T15:12:38.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:12:38.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:12:38.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-08-19T15:12:38.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:12:38.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-08-19T15:12:38.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:12:38.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:12:38.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-08-19T15:12:38.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:12:38.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-08-19T15:12:38.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:12:38.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:12:38.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:12:38.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:12:38.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:12:38.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:12:38.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-08-19T15:12:38.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:12:38.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:12:38.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-08-19T15:12:38.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:12:38.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:12:38.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-08-19T15:12:40.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-08-19T15:12:40.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:12:40.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:12:40.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-19T15:12:40.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:12:40.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:12:40.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:12:40.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:12:40.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:12:40.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:12:40.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T15:12:40.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T15:12:40.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:12:40.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:12:40.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:12:40.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:12:40.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:12:40.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:12:40.919 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:12:40.919 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:12:40.919 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:12:40.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:12:40.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:12:40.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:12:40.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:12:40.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:12:40.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:12:40.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:12:40.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:12:40.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:12:40.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:12:40.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:12:40.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T15:12:40.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-19T15:12:40.920 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.2 2024-08-19T15:12:40.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:12:40.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-08-19T15:12:40.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-19T15:12:40.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:12:40.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:12:40.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:12:40.948 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:40.946+0000 7fb9ddc13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:12:40.950 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:40.949+0000 7fb9ddc13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:12:40.951 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:40.951+0000 7fb9ddc13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:12:41.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-19T15:12:41.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:12:41.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T15:12:41.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:12:41.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:12:41.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:41.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:12:41.201 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:12:41.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:41.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:12:41.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:12:41.476 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:41.476+0000 7fb9ddc13640 -1 Falling back to public interface 2024-08-19T15:12:42.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:12:42.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:42.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:12:42.468 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:12:42.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:42.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:12:42.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:12:43.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:12:43.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:43.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:12:43.762 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:12:43.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:43.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:12:43.790 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:43.789+0000 7fb9ddc13640 -1 osd.2 68 log_to_monitors true 2024-08-19T15:12:44.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:12:45.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:12:45.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:45.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:12:45.123 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:12:45.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:45.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:12:45.445 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 up in weight 1 up_from 73 up_thru 73 down_at 69 last_clean_interval [15,68) [v2:127.0.0.1:6818/295682505,v1:127.0.0.1:6819/295682505] [v2:127.0.0.1:6820/295682505,v1:127.0.0.1:6821/295682505] exists,up 8d5c26d9-bfaa-498e-b826-ea3d9adf813b 2024-08-19T15:12:45.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:12:45.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:12:45.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:12:45.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:12:45.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:12:45.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:12:45.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:12:45.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:12:45.447 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:12:45.447 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:12:45.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:12:45.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:12:45.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:12:45.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:12:45.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:12:45.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:12:45.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:12:45.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:12:45.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:12:45.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:12:45.806 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:12:45.806 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:12:45.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:12:45.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:12:45.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:12:45.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841541 2024-08-19T15:12:45.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841541 2024-08-19T15:12:45.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541' 2024-08-19T15:12:45.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:12:45.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:12:46.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135881 2024-08-19T15:12:46.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135881 2024-08-19T15:12:46.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541 1-244813135881' 2024-08-19T15:12:46.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:12:46.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:12:46.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612610 2024-08-19T15:12:46.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612610 2024-08-19T15:12:46.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541 1-244813135881 2-313532612610' 2024-08-19T15:12:46.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:12:46.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841541 2024-08-19T15:12:46.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:12:46.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:12:46.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841541 2024-08-19T15:12:46.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:12:46.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841541 2024-08-19T15:12:46.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841541' 2024-08-19T15:12:46.118 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 283467841541 2024-08-19T15:12:46.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:12:46.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841540 -lt 283467841541 2024-08-19T15:12:46.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:12:47.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:12:47.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:12:47.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841541 -lt 283467841541 2024-08-19T15:12:47.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:12:47.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135881 2024-08-19T15:12:47.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:12:47.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:12:47.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135881 2024-08-19T15:12:47.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:12:47.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135881 2024-08-19T15:12:47.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135881' 2024-08-19T15:12:47.667 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 244813135881 2024-08-19T15:12:47.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:12:47.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135881 -lt 244813135881 2024-08-19T15:12:47.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:12:47.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612610 2024-08-19T15:12:47.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:12:47.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:12:47.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612610 2024-08-19T15:12:47.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:12:47.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612610 2024-08-19T15:12:47.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612610' 2024-08-19T15:12:47.972 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 313532612610 2024-08-19T15:12:47.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:12:48.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612610 -lt 313532612610 2024-08-19T15:12:48.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:12:48.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:12:48.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:12:48.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:12:48.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:12:48.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:12:48.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:12:48.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:12:48.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:12:48.576 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:12:48.576 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:12:48.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:12:48.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:12:48.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:12:48.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:12:49.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:12:49.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:12:49.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:12:49.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-08-19T15:12:49.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-08-19T15:12:49.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:12:49.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:12:49.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-08-19T15:12:49.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:12:49.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-08-19T15:12:49.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:12:49.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:12:49.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-08-19T15:12:49.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:12:49.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-08-19T15:12:49.195 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:12:49.195 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:12:49.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:12:49.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:12:49.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:12:49.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:12:49.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-08-19T15:12:49.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:12:49.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:12:49.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-08-19T15:12:49.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:12:49.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:12:49.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-08-19T15:12:51.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-08-19T15:12:51.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:12:51.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:12:51.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-19T15:12:51.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:12:51.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:12:51.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:12:51.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:12:51.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:12:51.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:12:51.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T15:12:51.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T15:12:51.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:12:51.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:12:51.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:12:51.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:12:51.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:12:51.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:12:51.668 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:12:51.668 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:12:51.668 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:12:51.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:12:51.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:12:51.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:12:51.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:12:51.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:12:51.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:12:51.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:12:51.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:12:51.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:12:51.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:12:51.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:12:51.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T15:12:51.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-19T15:12:51.670 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.2 2024-08-19T15:12:51.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:12:51.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-08-19T15:12:51.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-19T15:12:51.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:12:51.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:12:51.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:12:51.698 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:51.696+0000 7fa2165da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:12:51.700 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:51.698+0000 7fa2165da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:12:51.702 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:51.700+0000 7fa2165da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:12:51.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-19T15:12:51.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:12:51.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T15:12:51.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:12:51.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:12:51.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:51.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:12:51.949 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:12:51.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:51.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:12:52.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:12:52.228 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:52.226+0000 7fa2165da640 -1 Falling back to public interface 2024-08-19T15:12:53.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:12:53.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:53.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:12:53.215 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:12:53.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:53.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:12:53.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:12:54.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:12:54.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:54.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:12:54.490 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:12:54.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:54.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:12:54.626 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:12:54.625+0000 7fa2165da640 -1 osd.2 75 log_to_monitors true 2024-08-19T15:12:54.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:12:55.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:12:55.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:12:55.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:12:55.783 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:12:55.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:12:55.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:12:56.196 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 up in weight 1 up_from 80 up_thru 80 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6818/3605641120,v1:127.0.0.1:6819/3605641120] [v2:127.0.0.1:6820/3605641120,v1:127.0.0.1:6821/3605641120] exists,up 8d5c26d9-bfaa-498e-b826-ea3d9adf813b 2024-08-19T15:12:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:12:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:12:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:12:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:12:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:12:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:12:56.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:12:56.197 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:12:56.198 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:12:56.198 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:12:56.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:12:56.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:12:56.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:12:56.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:12:56.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:12:56.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:12:56.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:12:56.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:12:56.290 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:12:56.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:12:56.625 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:12:56.626 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:12:56.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:12:56.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:12:56.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:12:56.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841544 2024-08-19T15:12:56.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841544 2024-08-19T15:12:56.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841544' 2024-08-19T15:12:56.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:12:56.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:12:56.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135884 2024-08-19T15:12:56.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135884 2024-08-19T15:12:56.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841544 1-244813135884' 2024-08-19T15:12:56.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:12:56.834 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:12:56.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383682 2024-08-19T15:12:56.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383682 2024-08-19T15:12:56.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841544 1-244813135884 2-343597383682' 2024-08-19T15:12:56.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:12:56.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841544 2024-08-19T15:12:56.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:12:56.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:12:56.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841544 2024-08-19T15:12:56.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:12:56.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841544 2024-08-19T15:12:56.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841544' 2024-08-19T15:12:56.937 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 283467841544 2024-08-19T15:12:56.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:12:57.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841543 -lt 283467841544 2024-08-19T15:12:57.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:12:58.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:12:58.202 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:12:58.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841543 -lt 283467841544 2024-08-19T15:12:58.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:12:59.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:12:59.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:12:59.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841545 -lt 283467841544 2024-08-19T15:12:59.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:12:59.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135884 2024-08-19T15:12:59.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:12:59.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:12:59.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135884 2024-08-19T15:12:59.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:12:59.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135884 2024-08-19T15:12:59.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135884' 2024-08-19T15:12:59.758 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 244813135884 2024-08-19T15:12:59.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:13:00.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135884 -lt 244813135884 2024-08-19T15:13:00.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:13:00.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383682 2024-08-19T15:13:00.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:13:00.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:13:00.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383682 2024-08-19T15:13:00.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:13:00.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383682 2024-08-19T15:13:00.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383682' 2024-08-19T15:13:00.029 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 343597383682 2024-08-19T15:13:00.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:13:00.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383682 -lt 343597383682 2024-08-19T15:13:00.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:13:00.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:13:00.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:13:00.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:13:00.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:13:00.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:13:00.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:13:00.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:13:00.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:13:00.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:13:00.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:13:00.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:13:00.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:13:00.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:13:00.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:13:01.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:13:01.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:13:01.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:13:01.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-08-19T15:13:01.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-08-19T15:13:01.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-08-19T15:13:01.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-08-19T15:13:01.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:13:01.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:13:01.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-08-19T15:13:01.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:13:01.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:13:01.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:13:01.659 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:13:01.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:13:02.035 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:13:02.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:13:02.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:13:02.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:13:02.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-08-19T15:13:02.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-08-19T15:13:02.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-08-19T15:13:02.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-08-19T15:13:02.292 INFO:tasks.workunit.client.0.smithi102.stderr:2+0 records in 2024-08-19T15:13:02.292 INFO:tasks.workunit.client.0.smithi102.stderr:2+0 records out 2024-08-19T15:13:02.293 INFO:tasks.workunit.client.0.smithi102.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 7.0676e-05 s, 58.0 MB/s 2024-08-19T15:13:02.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:13:02.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:13:02.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:13:02.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:13:02.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:13:02.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:13:02.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:13:02.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:13:02.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:13:02.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:13:02.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:13:02.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:13:02.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:13:02.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:13:02.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:13:02.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:13:02.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:13:02.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:13:02.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:13:02.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:13:02.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:13:02.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:13:02.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:13:02.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:13:04.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:13:04.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:13:04.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:13:04.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:13:04.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:13:04.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:13:04.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:13:04.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:13:04.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:13:04.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:13:04.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:13:04.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:13:04.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:13:04.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:13:04.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:13:04.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:13:04.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:13:04.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:13:04.743 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:13:04.743 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:13:04.743 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:13:04.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:13:04.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:13:04.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:13:04.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:13:04.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:13:04.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:13:04.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:13:04.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:13:04.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:13:04.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:13:04.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:13:04.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:13:04.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:13:04.745 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:13:04.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:13:04.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:13:04.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:13:04.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:13:04.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:13:04.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:13:04.772 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:04.771+0000 7fb983adb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:13:04.775 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:04.774+0000 7fb983adb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:13:04.776 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:04.776+0000 7fb983adb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:13:05.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:13:05.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:13:05.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:13:05.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:13:05.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:13:05.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:05.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:13:05.026 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:13:05.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:05.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:13:05.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:13:05.304 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:05.303+0000 7fb983adb640 -1 Falling back to public interface 2024-08-19T15:13:06.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:13:06.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:06.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:13:06.291 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:13:06.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:06.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:13:06.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:13:07.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:13:07.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:07.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:13:07.569 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:13:07.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:07.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:13:07.680 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:07.679+0000 7fb983adb640 -1 osd.1 84 log_to_monitors true 2024-08-19T15:13:07.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:13:08.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:13:08.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:08.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:13:08.952 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:13:08.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:08.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:13:09.315 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 89 up_thru 89 down_at 85 last_clean_interval [57,84) [v2:127.0.0.1:6810/1584979963,v1:127.0.0.1:6811/1584979963] [v2:127.0.0.1:6812/1584979963,v1:127.0.0.1:6813/1584979963] exists,up c78dc1f0-99d1-4c82-8258-f34876d36832 2024-08-19T15:13:09.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:13:09.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:13:09.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:13:09.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:13:09.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:13:09.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:13:09.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:13:09.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:13:09.317 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:13:09.317 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:13:09.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:13:09.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:13:09.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:13:09.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:13:09.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:13:09.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:13:09.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:13:09.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:13:09.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:13:09.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:13:09.705 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:13:09.705 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:13:09.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:13:09.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:13:09.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:13:09.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841548 2024-08-19T15:13:09.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841548 2024-08-19T15:13:09.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841548' 2024-08-19T15:13:09.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:13:09.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:13:09.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089346 2024-08-19T15:13:09.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089346 2024-08-19T15:13:09.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841548 1-382252089346' 2024-08-19T15:13:09.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:13:09.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:13:10.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383686 2024-08-19T15:13:10.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383686 2024-08-19T15:13:10.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841548 1-382252089346 2-343597383686' 2024-08-19T15:13:10.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:13:10.013 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841548 2024-08-19T15:13:10.013 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:13:10.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:13:10.014 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841548 2024-08-19T15:13:10.014 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:13:10.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841548 2024-08-19T15:13:10.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841548' 2024-08-19T15:13:10.015 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 283467841548 2024-08-19T15:13:10.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:13:10.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841547 -lt 283467841548 2024-08-19T15:13:10.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:13:11.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:13:11.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:13:11.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841548 -lt 283467841548 2024-08-19T15:13:11.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:13:11.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-382252089346 2024-08-19T15:13:11.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:13:11.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:13:11.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-382252089346 2024-08-19T15:13:11.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:13:11.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089346 2024-08-19T15:13:11.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 382252089346' 2024-08-19T15:13:11.558 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 382252089346 2024-08-19T15:13:11.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:13:11.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089346 -lt 382252089346 2024-08-19T15:13:11.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:13:11.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383686 2024-08-19T15:13:11.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:13:11.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:13:11.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383686 2024-08-19T15:13:11.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:13:11.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383686 2024-08-19T15:13:11.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383686' 2024-08-19T15:13:11.830 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 343597383686 2024-08-19T15:13:11.830 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:13:12.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383686 -lt 343597383686 2024-08-19T15:13:12.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:13:12.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:13:12.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:13:12.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:13:12.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:13:12.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:13:12.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:13:12.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:13:12.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:13:12.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:13:12.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:13:12.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:13:12.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:13:12.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:13:12.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:13:13.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:13:13.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:13:13.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:13:13.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-08-19T15:13:13.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-08-19T15:13:13.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-08-19T15:13:13.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:13:13.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:13:13.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-08-19T15:13:13.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:13:13.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:13:13.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:13:13.407 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:13:13.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:13:13.877 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:13:13.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:13:13.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:13:13.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:13:14.047 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-08-19T15:13:14.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-08-19T15:13:14.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-08-19T15:13:14.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-08-19T15:13:14.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:13:14.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:13:14.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:13:14.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:13:14.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-08-19T15:13:14.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:13:14.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:13:14.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:13:14.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:13:14.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:13:14.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:13:14.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:13:14.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:13:14.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:13:14.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:13:14.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:13:14.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-08-19T15:13:14.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:13:14.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:13:14.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:13:14.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:13:14.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:13:14.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-08-19T15:13:16.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:13:16.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:13:16.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:13:16.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:13:16.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:13:16.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:13:16.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:13:16.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:13:16.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:13:16.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:13:16.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:13:16.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:13:16.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:13:16.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:13:16.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:13:16.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:13:16.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:13:16.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:13:16.434 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:13:16.434 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:13:16.434 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:13:16.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:13:16.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:13:16.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:13:16.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:13:16.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:13:16.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:13:16.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:13:16.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:13:16.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:13:16.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:13:16.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:13:16.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:13:16.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:13:16.436 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:13:16.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:13:16.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:13:16.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:13:16.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:13:16.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:13:16.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:13:16.463 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:16.462+0000 7f82c880d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:13:16.467 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:16.466+0000 7f82c880d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:13:16.468 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:16.467+0000 7f82c880d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:13:16.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:13:16.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:13:16.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:13:16.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:13:16.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:13:16.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:16.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:13:16.712 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:13:16.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:16.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:13:16.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:13:16.989 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:16.989+0000 7f82c880d640 -1 Falling back to public interface 2024-08-19T15:13:17.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:13:17.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:17.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:13:17.976 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:13:17.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:17.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:13:18.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:13:19.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:13:19.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:19.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:13:19.243 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:13:19.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:19.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:13:19.288 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:19.287+0000 7f82c880d640 -1 osd.0 93 log_to_monitors true 2024-08-19T15:13:19.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:13:20.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:13:20.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:20.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:13:20.578 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:13:20.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:20.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:13:20.911 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 98 up_thru 98 down_at 94 last_clean_interval [66,93) [v2:127.0.0.1:6802/1420084,v1:127.0.0.1:6803/1420084] [v2:127.0.0.1:6804/1420084,v1:127.0.0.1:6805/1420084] exists,up 791bfa96-95df-4f85-9b52-0573c1cfef78 2024-08-19T15:13:20.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:13:20.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:13:20.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:13:20.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:13:20.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:13:20.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:13:20.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:13:20.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:13:20.912 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:13:20.912 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:13:20.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:13:20.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:13:20.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:13:21.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:13:21.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:13:21.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:13:21.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:13:21.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:13:21.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:13:21.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:13:21.388 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:13:21.388 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:13:21.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:13:21.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:13:21.388 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:13:21.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795010 2024-08-19T15:13:21.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795010 2024-08-19T15:13:21.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010' 2024-08-19T15:13:21.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:13:21.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:13:21.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089349 2024-08-19T15:13:21.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089349 2024-08-19T15:13:21.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010 1-382252089349' 2024-08-19T15:13:21.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:13:21.623 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:13:21.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383689 2024-08-19T15:13:21.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383689 2024-08-19T15:13:21.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010 1-382252089349 2-343597383689' 2024-08-19T15:13:21.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:13:21.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-420906795010 2024-08-19T15:13:21.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:13:21.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:13:21.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-420906795010 2024-08-19T15:13:21.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:13:21.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795010 2024-08-19T15:13:21.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 420906795010' 2024-08-19T15:13:21.728 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 420906795010 2024-08-19T15:13:21.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:13:21.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 420906795010 2024-08-19T15:13:21.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:13:22.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:13:22.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:13:23.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795010 -lt 420906795010 2024-08-19T15:13:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:13:23.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-382252089349 2024-08-19T15:13:23.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:13:23.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:13:23.269 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-382252089349 2024-08-19T15:13:23.269 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:13:23.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089349 2024-08-19T15:13:23.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 382252089349' 2024-08-19T15:13:23.270 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 382252089349 2024-08-19T15:13:23.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:13:23.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089349 -lt 382252089349 2024-08-19T15:13:23.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:13:23.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383689 2024-08-19T15:13:23.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:13:23.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:13:23.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383689 2024-08-19T15:13:23.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:13:23.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383689 2024-08-19T15:13:23.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383689' 2024-08-19T15:13:23.541 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 343597383689 2024-08-19T15:13:23.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:13:23.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383689 -lt 343597383689 2024-08-19T15:13:23.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:13:23.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:13:23.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:13:24.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:13:24.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:13:24.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:13:24.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:13:24.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:13:24.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:13:24.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:13:24.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:13:24.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:13:24.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:13:24.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:13:24.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:13:24.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:13:24.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:13:24.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:13:24.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-08-19T15:13:24.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-08-19T15:13:24.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:13:24.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:13:24.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:13:24.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:13:24.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-08-19T15:13:24.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:13:24.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:13:24.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:13:24.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:13:24.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:13:24.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:13:24.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:13:24.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:13:24.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:13:24.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:13:25.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:13:25.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-08-19T15:13:25.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:13:25.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:13:25.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:13:25.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:13:25.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:13:25.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-08-19T15:13:27.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:13:27.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:13:27.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:13:27.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:13:27.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:13:27.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:13:27.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:13:27.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:13:27.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:13:27.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:13:27.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:13:27.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:13:27.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:13:27.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:13:27.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:13:27.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:13:27.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:13:27.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:13:27.055 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:13:27.055 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:13:27.056 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:13:27.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:13:27.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:13:27.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:13:27.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:13:27.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:13:27.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:13:27.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:13:27.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:13:27.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:13:27.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:13:27.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:13:27.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:13:27.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:13:27.058 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:13:27.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:13:27.058 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:13:27.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:13:27.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:13:27.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:13:27.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:13:27.085 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:27.084+0000 7f5431aff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:13:27.087 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:27.086+0000 7f5431aff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:13:27.089 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:27.088+0000 7f5431aff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:13:27.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:13:27.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:13:27.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:13:27.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:13:27.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:13:27.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:27.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:13:27.331 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:13:27.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:27.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:13:27.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:13:27.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:27.609+0000 7f5431aff640 -1 Falling back to public interface 2024-08-19T15:13:28.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:13:28.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:28.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:13:28.599 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:13:28.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:28.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:13:28.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:13:29.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:13:29.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:29.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:13:29.875 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:13:29.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:29.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:13:29.976 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:29.975+0000 7f5431aff640 -1 osd.1 101 log_to_monitors true 2024-08-19T15:13:30.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:13:31.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:13:31.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:31.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:13:31.147 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:13:31.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:31.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:13:31.475 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 106 up_thru 106 down_at 102 last_clean_interval [89,101) [v2:127.0.0.1:6810/1750529831,v1:127.0.0.1:6811/1750529831] [v2:127.0.0.1:6812/1750529831,v1:127.0.0.1:6813/1750529831] exists,up c78dc1f0-99d1-4c82-8258-f34876d36832 2024-08-19T15:13:31.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:13:31.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:13:31.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:13:31.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:13:31.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:13:31.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:13:31.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:13:31.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:13:31.476 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:13:31.476 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:13:31.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:13:31.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:13:31.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:13:31.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:13:31.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:13:31.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:13:31.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:13:31.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:13:31.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:13:31.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:13:31.854 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:13:31.854 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:13:31.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:13:31.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:13:31.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:13:31.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795013 2024-08-19T15:13:31.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795013 2024-08-19T15:13:31.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795013' 2024-08-19T15:13:31.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:13:31.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:13:32.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533378 2024-08-19T15:13:32.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533378 2024-08-19T15:13:32.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795013 1-455266533378' 2024-08-19T15:13:32.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:13:32.058 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:13:32.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383692 2024-08-19T15:13:32.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383692 2024-08-19T15:13:32.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795013 1-455266533378 2-343597383692' 2024-08-19T15:13:32.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:13:32.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-420906795013 2024-08-19T15:13:32.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:13:32.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:13:32.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-420906795013 2024-08-19T15:13:32.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:13:32.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795013 2024-08-19T15:13:32.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 420906795013' 2024-08-19T15:13:32.164 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 420906795013 2024-08-19T15:13:32.164 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:13:32.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795012 -lt 420906795013 2024-08-19T15:13:32.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:13:33.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:13:33.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:13:33.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795013 -lt 420906795013 2024-08-19T15:13:33.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:13:33.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533378 2024-08-19T15:13:33.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:13:33.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:13:33.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533378 2024-08-19T15:13:33.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:13:33.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533378 2024-08-19T15:13:33.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533378' 2024-08-19T15:13:33.709 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 455266533378 2024-08-19T15:13:33.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:13:33.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533378 -lt 455266533378 2024-08-19T15:13:33.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:13:33.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383692 2024-08-19T15:13:33.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:13:33.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:13:33.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383692 2024-08-19T15:13:33.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:13:33.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383692 2024-08-19T15:13:33.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383692' 2024-08-19T15:13:33.978 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 343597383692 2024-08-19T15:13:33.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:13:34.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383692 -lt 343597383692 2024-08-19T15:13:34.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:13:34.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:13:34.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:13:34.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:13:34.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:13:34.573 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:13:34.573 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:13:34.574 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:13:34.574 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:13:34.574 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:13:34.574 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:13:34.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:13:34.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:13:34.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:13:34.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:13:35.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:13:35.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:13:35.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:13:35.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-08-19T15:13:35.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-08-19T15:13:35.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-08-19T15:13:35.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:13:35.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:13:35.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-08-19T15:13:35.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:13:35.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:13:35.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:13:35.551 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:13:35.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:13:36.054 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:13:36.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:13:36.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:13:36.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:13:36.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-08-19T15:13:36.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-08-19T15:13:36.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-08-19T15:13:36.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-08-19T15:13:36.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-08-19T15:13:36.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-08-19T15:13:36.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-08-19T15:13:36.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:13:36.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:13:36.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:13:36.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:13:36.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-08-19T15:13:36.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:13:36.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:13:36.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:13:36.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:13:36.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:13:36.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:13:36.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:13:36.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:13:36.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:13:36.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:13:36.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:13:36.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-08-19T15:13:36.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:13:36.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:13:36.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:13:36.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:13:36.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:13:36.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-08-19T15:13:38.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:13:38.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:13:38.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:13:38.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:13:38.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:13:38.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:13:38.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:13:38.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:13:38.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:13:38.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:13:38.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:13:38.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:13:38.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:13:38.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:13:38.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:13:38.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:13:38.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:13:38.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:13:38.721 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:13:38.721 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:13:38.721 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:13:38.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:13:38.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:13:38.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:13:38.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:13:38.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:13:38.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:13:38.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:13:38.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:13:38.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:13:38.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:13:38.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:13:38.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:13:38.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:13:38.724 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:13:38.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:13:38.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:13:38.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:13:38.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:13:38.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:13:38.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:13:38.751 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:38.749+0000 7f2353b88640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:13:38.753 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:38.751+0000 7f2353b88640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:13:38.754 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:38.752+0000 7f2353b88640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:13:38.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:13:39.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:13:39.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:13:39.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:13:39.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:13:39.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:39.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:13:39.000 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:13:39.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:39.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:13:39.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:13:39.283 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:39.281+0000 7f2353b88640 -1 Falling back to public interface 2024-08-19T15:13:40.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:13:40.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:40.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:13:40.266 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:13:40.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:40.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:13:40.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:13:41.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:13:41.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:41.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:13:41.552 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:13:41.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:41.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:13:41.622 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:41.620+0000 7f2353b88640 -1 osd.0 110 log_to_monitors true 2024-08-19T15:13:41.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:13:42.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:13:42.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:42.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:13:42.889 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:13:42.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:42.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:13:43.273 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 115 up_thru 115 down_at 111 last_clean_interval [98,110) [v2:127.0.0.1:6802/2191351443,v1:127.0.0.1:6803/2191351443] [v2:127.0.0.1:6804/2191351443,v1:127.0.0.1:6805/2191351443] exists,up 791bfa96-95df-4f85-9b52-0573c1cfef78 2024-08-19T15:13:43.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:13:43.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:13:43.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:13:43.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:13:43.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:13:43.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:13:43.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:13:43.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:13:43.274 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:13:43.274 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:13:43.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:13:43.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:13:43.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:13:43.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:13:43.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:13:43.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:13:43.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:13:43.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:13:43.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:13:43.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:13:43.676 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:13:43.676 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:13:43.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:13:43.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:13:43.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:13:43.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=493921239042 2024-08-19T15:13:43.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 493921239042 2024-08-19T15:13:43.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-493921239042' 2024-08-19T15:13:43.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:13:43.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:13:43.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533381 2024-08-19T15:13:43.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533381 2024-08-19T15:13:43.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-493921239042 1-455266533381' 2024-08-19T15:13:43.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:13:43.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:13:43.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383695 2024-08-19T15:13:43.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383695 2024-08-19T15:13:43.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-493921239042 1-455266533381 2-343597383695' 2024-08-19T15:13:43.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:13:43.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-493921239042 2024-08-19T15:13:43.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:13:43.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:13:43.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-493921239042 2024-08-19T15:13:43.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:13:43.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=493921239042 2024-08-19T15:13:43.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 493921239042' 2024-08-19T15:13:43.987 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 493921239042 2024-08-19T15:13:43.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:13:44.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 493921239042 2024-08-19T15:13:44.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:13:45.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:13:45.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:13:45.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 493921239042 -lt 493921239042 2024-08-19T15:13:45.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:13:45.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533381 2024-08-19T15:13:45.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:13:45.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:13:45.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533381 2024-08-19T15:13:45.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:13:45.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533381 2024-08-19T15:13:45.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533381' 2024-08-19T15:13:45.531 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 455266533381 2024-08-19T15:13:45.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:13:45.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533381 -lt 455266533381 2024-08-19T15:13:45.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:13:45.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383695 2024-08-19T15:13:45.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:13:45.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:13:45.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383695 2024-08-19T15:13:45.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:13:45.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383695 2024-08-19T15:13:45.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383695' 2024-08-19T15:13:45.800 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 343597383695 2024-08-19T15:13:45.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:13:46.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383695 -lt 343597383695 2024-08-19T15:13:46.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:13:46.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:13:46.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:13:46.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:13:46.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:13:46.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:13:46.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:13:46.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:13:46.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:13:46.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:13:46.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:13:46.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:13:46.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:13:46.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:13:46.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:13:47.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:13:47.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:13:47.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:13:47.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-08-19T15:13:47.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:13:47.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:13:47.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:13:47.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:13:47.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-08-19T15:13:47.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:13:47.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:13:47.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:13:47.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:13:47.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:13:47.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:13:47.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:13:47.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:13:47.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:13:47.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:13:47.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:13:47.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-08-19T15:13:47.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:13:47.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:13:47.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:13:47.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:13:47.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:13:47.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-08-19T15:13:49.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:13:49.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:13:49.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:13:49.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:13:49.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:13:49.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:13:49.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:13:49.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:13:49.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:13:49.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:13:49.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:13:49.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:13:49.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:13:49.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:13:49.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:13:49.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:13:49.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:13:49.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:13:49.791 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:13:49.791 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:13:49.791 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:13:49.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:13:49.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:13:49.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:13:49.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:13:49.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:13:49.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:13:49.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:13:49.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:13:49.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:13:49.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:13:49.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:13:49.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:13:49.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:13:49.793 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:13:49.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:13:49.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:13:49.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:13:49.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:13:49.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:13:49.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:13:49.821 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:49.820+0000 7f78d720d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:13:49.823 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:49.822+0000 7f78d720d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:13:49.825 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:49.824+0000 7f78d720d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:13:50.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:13:50.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:13:50.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:13:50.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:13:50.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:13:50.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:50.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:13:50.079 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:13:50.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:50.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:13:50.343 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:50.342+0000 7f78d720d640 -1 Falling back to public interface 2024-08-19T15:13:50.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:13:51.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:13:51.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:51.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:13:51.389 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:13:51.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:51.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:13:51.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:13:52.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:13:52.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:52.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:13:52.663 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:13:52.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:52.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:13:52.771 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:13:52.770+0000 7f78d720d640 -1 osd.0 117 log_to_monitors true 2024-08-19T15:13:52.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:13:53.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:13:53.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:13:53.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:13:53.976 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:13:53.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:13:53.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:13:54.318 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 122 up_thru 122 down_at 118 last_clean_interval [115,117) [v2:127.0.0.1:6802/2802716815,v1:127.0.0.1:6803/2802716815] [v2:127.0.0.1:6804/2802716815,v1:127.0.0.1:6805/2802716815] exists,up 791bfa96-95df-4f85-9b52-0573c1cfef78 2024-08-19T15:13:54.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:13:54.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:13:54.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:13:54.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:13:54.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:13:54.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:13:54.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:13:54.319 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:13:54.320 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:13:54.320 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:13:54.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:13:54.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:13:54.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:13:54.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:13:54.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:13:54.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:13:54.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:13:54.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:13:54.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:13:54.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:13:54.692 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:13:54.692 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:13:54.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:13:54.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:13:54.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:13:54.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=523986010114 2024-08-19T15:13:54.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 523986010114 2024-08-19T15:13:54.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114' 2024-08-19T15:13:54.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:13:54.810 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:13:54.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533384 2024-08-19T15:13:54.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533384 2024-08-19T15:13:54.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114 1-455266533384' 2024-08-19T15:13:54.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:13:54.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:13:55.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383699 2024-08-19T15:13:55.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383699 2024-08-19T15:13:55.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114 1-455266533384 2-343597383699' 2024-08-19T15:13:55.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:13:55.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-523986010114 2024-08-19T15:13:55.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:13:55.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:13:55.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-523986010114 2024-08-19T15:13:55.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:13:55.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=523986010114 2024-08-19T15:13:55.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 523986010114' 2024-08-19T15:13:55.018 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 523986010114 2024-08-19T15:13:55.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:13:55.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 523986010114 2024-08-19T15:13:55.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:13:56.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:13:56.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:13:56.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 523986010114 -lt 523986010114 2024-08-19T15:13:56.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:13:56.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533384 2024-08-19T15:13:56.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:13:56.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:13:56.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533384 2024-08-19T15:13:56.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:13:56.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533384 2024-08-19T15:13:56.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533384' 2024-08-19T15:13:56.564 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 455266533384 2024-08-19T15:13:56.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:13:56.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533385 -lt 455266533384 2024-08-19T15:13:56.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:13:56.834 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383699 2024-08-19T15:13:56.834 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:13:56.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:13:56.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383699 2024-08-19T15:13:56.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:13:56.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383699 2024-08-19T15:13:56.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383699' 2024-08-19T15:13:56.837 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.2 seq 343597383699 2024-08-19T15:13:56.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:13:57.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383699 -lt 343597383699 2024-08-19T15:13:57.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:13:57.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:13:57.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:13:57.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:13:57.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:13:57.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:13:57.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:13:57.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:13:57.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:13:57.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:13:57.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:13:57.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:13:57.698 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:13:57.698 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:13:57.698 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:13:58.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:13:58.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:13:58.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:13:58.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-08-19T15:13:58.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-08-19T15:13:58.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T15:13:58.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-08-19T15:13:58.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-08-19T15:13:58.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:13:58.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-08-19T15:13:58.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-08-19T15:13:58.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-08-19T15:13:58.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-08-19T15:13:58.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:13:58.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:13:58.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:13:58.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:13:58.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-08-19T15:11:41.982413+0000 2024-08-19T15:13:58.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-08-19T15:13:58.831 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-08-19T15:13:58.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-08-19T15:11:41.982413+0000 2024-08-19T15:13:58.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-08-19T15:13:58.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:11:41.982413+0000 2024-08-19T15:13:58.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:13:58.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:13:58.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:13:58.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-08-19T15:13:58.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:13:58.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:13:58.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:13:58.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:13:59.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:11:41.982413+0000 '>' 2024-08-19T15:11:41.982413+0000 2024-08-19T15:13:59.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:14:00.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:14:00.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:14:00.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-08-19T15:14:00.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:14:00.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:14:00.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:14:00.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:14:00.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:11:41.982413+0000 '>' 2024-08-19T15:11:41.982413+0000 2024-08-19T15:14:00.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:14:01.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:14:01.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:14:01.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-08-19T15:14:01.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:14:01.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:14:01.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:14:01.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:14:01.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:13:59.087676+0000 '>' 2024-08-19T15:11:41.982413+0000 2024-08-19T15:14:01.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:14:01.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-08-19T15:14:01.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-08-19T15:14:01.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-08-19T15:14:01.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-08-19T15:14:01.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-08-19T15:14:01.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-08-19T15:14:01.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-08-19T15:14:01.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=122 2024-08-19T15:14:01.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-08-19T15:14:01.717 INFO:tasks.workunit.client.0.smithi102.stderr: . as $in 2024-08-19T15:14:01.718 INFO:tasks.workunit.client.0.smithi102.stderr: | if type == "object" then 2024-08-19T15:14:01.718 INFO:tasks.workunit.client.0.smithi102.stderr: reduce keys[] as $key 2024-08-19T15:14:01.718 INFO:tasks.workunit.client.0.smithi102.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-08-19T15:14:01.718 INFO:tasks.workunit.client.0.smithi102.stderr: elif type == "array" then map( walk(f) ) | f 2024-08-19T15:14:01.718 INFO:tasks.workunit.client.0.smithi102.stderr: else f 2024-08-19T15:14:01.718 INFO:tasks.workunit.client.0.smithi102.stderr: end; 2024-08-19T15:14:01.718 INFO:tasks.workunit.client.0.smithi102.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-08-19T15:14:01.718 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-08-19T15:14:01.718 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-08-19T15:14:01.718 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.version) else . end) 2024-08-19T15:14:01.718 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-08-19T15:14:01.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-08-19T15:14:01.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-08-19T15:14:01.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-08-19T15:14:01.741 INFO:tasks.workunit.client.0.smithi102.stderr: . as $in 2024-08-19T15:14:01.741 INFO:tasks.workunit.client.0.smithi102.stderr: | if type == "object" then 2024-08-19T15:14:01.741 INFO:tasks.workunit.client.0.smithi102.stderr: reduce keys[] as $key 2024-08-19T15:14:01.741 INFO:tasks.workunit.client.0.smithi102.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-08-19T15:14:01.741 INFO:tasks.workunit.client.0.smithi102.stderr: elif type == "array" then map( walk(f) ) | f 2024-08-19T15:14:01.741 INFO:tasks.workunit.client.0.smithi102.stderr: else f 2024-08-19T15:14:01.742 INFO:tasks.workunit.client.0.smithi102.stderr: end; 2024-08-19T15:14:01.742 INFO:tasks.workunit.client.0.smithi102.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-08-19T15:14:01.742 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-08-19T15:14:01.742 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-08-19T15:14:01.742 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.version) else . end) 2024-08-19T15:14:01.742 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-08-19T15:14:01.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-08-19T15:14:01.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-08-19T15:14:01.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-08-19T15:14:01.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-08-19T15:14:01.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-08-19T15:14:01.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-08-19T15:14:01.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-08-19T15:14:01.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-08-19T15:14:01.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:14:01.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:14:01.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:14:01.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:14:01.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:14:02.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-08-19T15:11:41.982413+0000 2024-08-19T15:14:02.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-08-19T15:14:02.278 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-08-19T15:14:02.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-08-19T15:11:41.982413+0000 last_deep_scrub_stamp 2024-08-19T15:14:02.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-08-19T15:14:02.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:11:41.982413+0000 2024-08-19T15:14:02.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-08-19T15:14:02.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:14:02.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:14:02.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:14:02.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:14:02.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:14:02.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:14:02.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:14:02.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:11:41.982413+0000 '>' 2024-08-19T15:11:41.982413+0000 2024-08-19T15:14:02.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:14:03.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:14:03.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:14:03.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:14:03.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:14:03.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:14:03.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:14:03.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:14:03.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:11:41.982413+0000 '>' 2024-08-19T15:11:41.982413+0000 2024-08-19T15:14:03.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:14:04.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:14:04.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:14:04.834 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:14:04.834 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:14:04.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:14:04.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:14:04.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:14:05.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:11:41.982413+0000 '>' 2024-08-19T15:11:41.982413+0000 2024-08-19T15:14:05.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:14:06.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:14:06.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:14:06.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:14:06.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:14:06.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:14:06.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:14:06.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:14:06.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:11:41.982413+0000 '>' 2024-08-19T15:11:41.982413+0000 2024-08-19T15:14:06.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:14:07.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:14:07.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:14:07.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:14:07.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:14:07.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:14:07.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:14:07.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:14:07.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:14:03.126840+0000 '>' 2024-08-19T15:11:41.982413+0000 2024-08-19T15:14:07.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:14:07.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-08-19T15:14:07.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-08-19T15:14:07.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-08-19T15:14:07.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-08-19T15:14:07.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-08-19T15:14:07.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-08-19T15:14:07.717 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-08-19T15:14:07.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=122 2024-08-19T15:14:07.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' true = true ']' 2024-08-19T15:14:07.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq 'def walk(f): 2024-08-19T15:14:07.720 INFO:tasks.workunit.client.0.smithi102.stderr: . as $in 2024-08-19T15:14:07.720 INFO:tasks.workunit.client.0.smithi102.stderr: | if type == "object" then 2024-08-19T15:14:07.721 INFO:tasks.workunit.client.0.smithi102.stderr: reduce keys[] as $key 2024-08-19T15:14:07.721 INFO:tasks.workunit.client.0.smithi102.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-08-19T15:14:07.721 INFO:tasks.workunit.client.0.smithi102.stderr: elif type == "array" then map( walk(f) ) | f 2024-08-19T15:14:07.721 INFO:tasks.workunit.client.0.smithi102.stderr: else f 2024-08-19T15:14:07.721 INFO:tasks.workunit.client.0.smithi102.stderr: end; 2024-08-19T15:14:07.721 INFO:tasks.workunit.client.0.smithi102.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-08-19T15:14:07.721 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-08-19T15:14:07.721 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-08-19T15:14:07.721 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.version) else . end) 2024-08-19T15:14:07.721 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-08-19T15:14:07.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq .inconsistents 2024-08-19T15:14:07.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-08-19T15:14:07.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-08-19T15:14:07.744 INFO:tasks.workunit.client.0.smithi102.stderr: . as $in 2024-08-19T15:14:07.744 INFO:tasks.workunit.client.0.smithi102.stderr: | if type == "object" then 2024-08-19T15:14:07.744 INFO:tasks.workunit.client.0.smithi102.stderr: reduce keys[] as $key 2024-08-19T15:14:07.744 INFO:tasks.workunit.client.0.smithi102.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-08-19T15:14:07.744 INFO:tasks.workunit.client.0.smithi102.stderr: elif type == "array" then map( walk(f) ) | f 2024-08-19T15:14:07.744 INFO:tasks.workunit.client.0.smithi102.stderr: else f 2024-08-19T15:14:07.744 INFO:tasks.workunit.client.0.smithi102.stderr: end; 2024-08-19T15:14:07.744 INFO:tasks.workunit.client.0.smithi102.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-08-19T15:14:07.744 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-08-19T15:14:07.744 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-08-19T15:14:07.744 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.version) else . end) 2024-08-19T15:14:07.745 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-08-19T15:14:07.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-08-19T15:14:07.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-08-19T15:14:07.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-08-19T15:14:07.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-08-19T15:14:07.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-08-19T15:14:07.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-08-19T15:14:07.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-08-19T15:14:08.121 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'ecpool' removed 2024-08-19T15:14:08.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T15:14:08.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:14:08.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:14:08.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:14:08.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:14:08.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:14:08.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:14:08.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:14:08.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:14:08.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:14:08.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:14:08.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:14:08.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:14:08.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:14:08.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:14:08.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:14:08.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:14:08.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:14:08.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:14:08.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:14:08.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:14:08.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:14:08.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:14:08.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:14:08.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:14:08.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:14:08.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:14:08.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:14:08.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:14:08.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T15:14:08.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T15:14:08.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T15:14:08.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T15:14:08.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:14:08.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:14:08.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:14:08.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:14:08.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:14:08.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:14:08.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:14:08.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:14:08.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:14:08.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:14:08.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:14:08.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:14:08.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:14:08.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:14:08.282 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:14:08.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:14:08.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:14:08.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:14:08.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:14:08.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:14:08.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:14:08.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:14:08.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:14:08.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:14:08.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:14:08.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:14:08.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:14:08.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:14:08.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T15:14:08.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T15:14:08.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:14:08.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:14:08.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T15:14:08.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T15:14:08.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T15:14:08.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T15:14:08.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T15:14:08.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2024-08-19T15:14:08.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2024-08-19T15:14:08.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:992: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2024-08-19T15:14:08.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:993: TEST_corrupt_scrub_replicated: local total_objs=19 2024-08-19T15:14:08.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-08-19T15:14:08.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T15:14:08.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T15:14:08.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T15:14:08.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T15:14:08.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T15:14:08.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-08-19T15:14:08.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T15:14:08.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:14:08.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:14:08.670 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:14:08.670 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:14:08.670 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:14:08.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:14:08.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-08-19T15:14:08.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T15:14:08.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T15:14:08.858 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:14:08.858 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:14:08.858 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T15:14:08.858 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T15:14:08.858 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:14:08.858 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:14:08.859 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:14:08.859 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:14:08.859 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:14:08.859 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:14:08.859 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:14:08.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:14:08.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T15:14:08.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T15:14:08.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:14:08.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:14:08.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T15:14:08.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T15:14:08.937 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:14:08.937 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:14:08.937 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:14:08.937 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:14:08.937 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:14:08.937 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:14:08.937 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:14:08.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:14:08.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T15:14:09.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:996: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-08-19T15:14:09.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T15:14:09.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T15:14:09.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T15:14:09.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T15:14:09.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T15:14:09.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T15:14:09.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T15:14:09.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:14:09.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:14:09.823 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:14:09.823 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:14:09.823 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:14:09.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:14:09.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:14:09.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:14:09.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:997: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2024-08-19T15:14:09.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:14:09.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:14:09.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T15:14:09.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:14:09.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:14:09.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:14:09.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:14:09.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:14:09.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:14:09.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:14:09.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:14:09.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:14:09.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:14:09.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:14:09.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:14:09.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:14:09.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:14:09.856 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:14:09.856 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:14:09.856 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:14:09.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:14:09.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:14:09.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:14:09.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:14:09.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:14:09.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:14:09.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:14:09.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:14:09.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:14:09.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:14:09.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:14:09.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:14:09.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:14:09.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:14:09.860 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:14:09.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a9a4af16-d461-4c2f-ab0d-b99df37a8a3e 2024-08-19T15:14:09.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a9a4af16-d461-4c2f-ab0d-b99df37a8a3e' 2024-08-19T15:14:09.861 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 a9a4af16-d461-4c2f-ab0d-b99df37a8a3e 2024-08-19T15:14:09.861 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:14:09.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBBYcNmNf4GNBAAe8ldilfisyCC+DHx8KLbIg== 2024-08-19T15:14:09.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBBYcNmNf4GNBAAe8ldilfisyCC+DHx8KLbIg=="}' 2024-08-19T15:14:09.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a9a4af16-d461-4c2f-ab0d-b99df37a8a3e -i td/osd-scrub-repair/0/new.json 2024-08-19T15:14:10.062 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:14:10.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T15:14:10.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBYcNmNf4GNBAAe8ldilfisyCC+DHx8KLbIg== --osd-uuid a9a4af16-d461-4c2f-ab0d-b99df37a8a3e 2024-08-19T15:14:10.101 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:10.100+0000 7fe9bb424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:14:10.104 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:10.104+0000 7fe9bb424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:14:10.106 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:10.105+0000 7fe9bb424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:14:10.106 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:10.105+0000 7fe9bb424640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:14:10.106 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:10.105+0000 7fe9bb424640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:14:10.106 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:10.105+0000 7fe9bb424640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:14:10.180 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:10.179+0000 7fe9bb424640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T15:14:15.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T15:14:15.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:14:15.551 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T15:14:15.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T15:14:15.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:14:15.894 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:14:15.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T15:14:15.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:14:15.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:14:15.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:14:15.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:14:15.920 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:15.919+0000 7fd98661b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:14:15.928 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:15.926+0000 7fd98661b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:14:15.928 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:15.927+0000 7fd98661b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:14:16.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T15:14:16.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:14:16.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:14:16.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:14:16.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:14:16.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:16.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:14:16.185 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:14:16.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:16.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:14:16.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:16.470 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:16.470+0000 7fd98661b640 -1 Falling back to public interface 2024-08-19T15:14:17.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:17.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:17.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:14:17.451 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:14:17.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:17.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:14:17.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:18.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:18.722 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:14:18.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:18.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:14:18.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:18.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:14:18.769 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:18.768+0000 7fd98661b640 -1 osd.0 0 log_to_monitors true 2024-08-19T15:14:18.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:20.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:20.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:20.000 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:14:20.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:14:20.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:20.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:14:20.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:21.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:21.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:21.276 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:14:21.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:14:21.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:21.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:14:21.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:22.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:22.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:22.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:14:22.640 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:14:22.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:22.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:14:22.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:23.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:23.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:23.923 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:14:23.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:14:23.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:23.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:14:24.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:25.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:25.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:25.257 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:14:25.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:14:25.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:25.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:14:25.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:26.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:26.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:26.626 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:14:26.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:14:26.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:26.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:14:26.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:27.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:27.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:27.904 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:14:27.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:14:27.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:27.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:14:28.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:29.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:29.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:29.179 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:14:29.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:14:29.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:29.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:14:29.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:30.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:30.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:30.456 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:14:30.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:14:30.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:30.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:14:31.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:32.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:32.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:32.037 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:14:32.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:14:32.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:32.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:14:32.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:33.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:33.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:33.580 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:14:33.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:14:33.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:33.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:14:34.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:35.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:35.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:35.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:14:35.399 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:14:35.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:35.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:14:36.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:37.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:37.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:37.016 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:14:37.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:14:37.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:37.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:14:37.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:38.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:38.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:38.637 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:14:38.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:14:38.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:38.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:14:39.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:40.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:40.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:40.242 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:14:40.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:14:40.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:40.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:14:40.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:41.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:41.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:41.731 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:14:41.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:14:41.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:41.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:14:42.010 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1154645348,v1:127.0.0.1:6803/1154645348] [v2:127.0.0.1:6804/1154645348,v1:127.0.0.1:6805/1154645348] exists,up a9a4af16-d461-4c2f-ab0d-b99df37a8a3e 2024-08-19T15:14:42.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:14:42.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:14:42.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:14:42.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2024-08-19T15:14:42.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:14:42.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:14:42.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T15:14:42.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:14:42.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:14:42.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:14:42.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:14:42.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:14:42.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:14:42.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:14:42.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:14:42.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:14:42.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:14:42.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:14:42.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:14:42.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:14:42.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:14:42.016 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:14:42.016 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:14:42.016 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:14:42.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:14:42.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:14:42.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:14:42.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:14:42.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:14:42.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:14:42.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:14:42.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:14:42.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:14:42.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:14:42.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:14:42.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:14:42.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:14:42.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:14:42.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:14:42.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:14:42.018 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:14:42.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b' 2024-08-19T15:14:42.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:14:42.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBiYcNmL9TiARAAX2cRWkWyrtOakKV/UF2qgQ== 2024-08-19T15:14:42.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBiYcNmL9TiARAAX2cRWkWyrtOakKV/UF2qgQ=="}' 2024-08-19T15:14:42.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b -i td/osd-scrub-repair/1/new.json 2024-08-19T15:14:42.391 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:14:42.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T15:14:42.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBiYcNmL9TiARAAX2cRWkWyrtOakKV/UF2qgQ== --osd-uuid 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:14:42.431 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:42.430+0000 7fc93d20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:14:42.475 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:42.474+0000 7fc93d20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:14:42.476 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:42.475+0000 7fc93d20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:14:42.477 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:42.475+0000 7fc93d20c640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:14:42.477 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:42.476+0000 7fc93d20c640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:14:42.477 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:42.476+0000 7fc93d20c640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:14:42.550 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:42.549+0000 7fc93d20c640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T15:14:48.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T15:14:48.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:14:48.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T15:14:48.217 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T15:14:48.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:14:48.563 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:14:48.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T15:14:48.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:14:48.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:14:48.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:14:48.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:14:48.591 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:48.590+0000 7f0e6b933640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:14:48.593 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:48.592+0000 7f0e6b933640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:14:48.594 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:48.593+0000 7f0e6b933640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:14:48.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T15:14:48.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:14:48.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:14:48.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:14:48.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:14:48.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:48.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:14:48.835 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:14:48.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:48.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:14:49.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:49.131 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:49.130+0000 7f0e6b933640 -1 Falling back to public interface 2024-08-19T15:14:50.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:50.101 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:14:50.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:50.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:14:50.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:50.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:14:50.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:51.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:51.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:51.369 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:14:51.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:14:51.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:51.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:14:51.443 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:14:51.442+0000 7f0e6b933640 -1 osd.1 0 log_to_monitors true 2024-08-19T15:14:51.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:52.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:52.646 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:14:52.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:52.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:14:52.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:52.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:14:52.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:53.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:53.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:53.973 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:14:53.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:14:53.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:53.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:14:54.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:55.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:55.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:55.282 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:14:55.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:14:55.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:55.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:14:55.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:56.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:56.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:56.558 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:14:56.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:14:56.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:56.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:14:56.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:57.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:57.833 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:14:57.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:57.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:14:57.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:57.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:14:58.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:14:59.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:14:59.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:14:59.218 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:14:59.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:14:59.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:14:59.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:14:59.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:15:00.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:15:00.494 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:15:00.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:15:00.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:15:00.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:15:00.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:15:00.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:15:01.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:15:01.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:15:01.774 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:15:01.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:15:01.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:15:01.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:15:02.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:15:03.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:15:03.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:15:03.070 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:15:03.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:15:03.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:15:03.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:15:03.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:15:04.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:15:04.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:15:04.650 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:15:04.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:15:04.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:15:04.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:15:05.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:15:06.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:15:06.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:15:06.032 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:15:06.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:15:06.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:15:06.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:15:06.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:15:07.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:15:07.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:15:07.725 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:15:07.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:15:07.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:15:07.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:15:08.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:15:09.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:15:09.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:15:09.176 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:15:09.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:15:09.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:15:09.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:15:09.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:15:10.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:15:10.770 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:15:10.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:15:10.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:15:10.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:15:10.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:15:11.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:15:12.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:15:12.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:15:12.530 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:15:12.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:15:12.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:15:12.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:15:13.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:15:14.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:15:14.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:15:14.189 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:15:14.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:15:14.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:15:14.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:15:14.458 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2895455250,v1:127.0.0.1:6811/2895455250] [v2:127.0.0.1:6812/2895455250,v1:127.0.0.1:6813/2895455250] exists,up 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:15:14.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:15:14.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:15:14.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:15:14.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: TEST_corrupt_scrub_replicated: create_rbd_pool 2024-08-19T15:15:14.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T15:15:14.734 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-08-19T15:15:14.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T15:15:14.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T15:15:15.115 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' created 2024-08-19T15:15:15.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:15:16.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T15:15:17.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1000: TEST_corrupt_scrub_replicated: wait_for_clean 2024-08-19T15:15:17.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:15:17.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:15:17.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:15:17.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:15:17.376 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:15:17.376 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:15:17.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:15:17.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:15:17.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:15:17.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:15:17.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:15:17.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:15:17.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:15:17.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:15:17.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:15:17.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:15:17.739 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:15:17.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:15:17.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:15:17.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:15:17.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836492 2024-08-19T15:15:17.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836492 2024-08-19T15:15:17.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492' 2024-08-19T15:15:17.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:15:17.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:15:17.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-08-19T15:15:17.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-08-19T15:15:17.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492 1-42949672970' 2024-08-19T15:15:17.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:15:17.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836492 2024-08-19T15:15:17.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:15:17.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:15:17.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836492 2024-08-19T15:15:17.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:15:17.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836492 2024-08-19T15:15:17.948 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836492 2024-08-19T15:15:17.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-08-19T15:15:17.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:15:18.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836492 2024-08-19T15:15:18.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:15:19.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:15:19.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:15:19.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836492 2024-08-19T15:15:19.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:15:20.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:15:20.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:15:20.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836492 2024-08-19T15:15:20.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:15:20.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-08-19T15:15:20.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:15:20.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:15:20.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-08-19T15:15:20.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:15:20.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-08-19T15:15:20.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-08-19T15:15:20.760 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672970 2024-08-19T15:15:20.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:15:21.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-08-19T15:15:21.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:15:21.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:15:21.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:15:21.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T15:15:21.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:15:21.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:15:21.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:15:21.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:15:21.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:15:21.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:15:21.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:15:21.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:15:21.701 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:15:21.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:15:21.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:15:22.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T15:15:22.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:15:22.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:15:22.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: TEST_corrupt_scrub_replicated: create_pool foo 1 2024-08-19T15:15:22.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-08-19T15:15:22.387 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'foo' created 2024-08-19T15:15:22.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:15:23.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2024-08-19T15:15:23.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-08-19T15:15:23.768 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'csr_pool' created 2024-08-19T15:15:23.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:15:24.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1004: TEST_corrupt_scrub_replicated: wait_for_clean 2024-08-19T15:15:24.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:15:24.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:15:24.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:15:24.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:15:24.783 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:15:24.783 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:15:24.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:15:24.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:15:24.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:15:24.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:15:24.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:15:24.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:15:24.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:15:24.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:15:24.889 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:15:25.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:15:25.175 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:15:25.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:15:25.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:15:25.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:15:25.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836494 2024-08-19T15:15:25.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836494 2024-08-19T15:15:25.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494' 2024-08-19T15:15:25.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:15:25.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:15:25.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-08-19T15:15:25.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-08-19T15:15:25.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494 1-42949672973' 2024-08-19T15:15:25.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:15:25.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836494 2024-08-19T15:15:25.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:15:25.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:15:25.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836494 2024-08-19T15:15:25.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:15:25.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836494 2024-08-19T15:15:25.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-08-19T15:15:25.428 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836494 2024-08-19T15:15:25.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:15:25.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836494 2024-08-19T15:15:25.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:15:26.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:15:26.698 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:15:26.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836494 2024-08-19T15:15:26.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:15:26.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-08-19T15:15:26.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:15:26.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:15:26.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-08-19T15:15:26.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:15:26.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-08-19T15:15:26.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-08-19T15:15:26.975 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672973 2024-08-19T15:15:26.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:15:27.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-08-19T15:15:27.238 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:15:27.238 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:15:27.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:15:27.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:15:27.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:15:27.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:15:27.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:15:27.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:15:27.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:15:27.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:15:27.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:15:27.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:15:27.830 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:15:27.831 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:15:27.831 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:15:28.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:15:28.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:15:28.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:15:28.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: seq 1 19 2024-08-19T15:15:28.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:15:28.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-08-19T15:15:28.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-08-19T15:15:28.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:15:28.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:15:28.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-08-19T15:15:28.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:15:28.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:15:28.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:15:28.522 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:15:28.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:15:29.033 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:15:29.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:15:29.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:15:29.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:15:29.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-08-19T15:15:29.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-08-19T15:15:29.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:15:29.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-08-19T15:15:29.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-08-19T15:15:29.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:15:29.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:15:29.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-08-19T15:15:29.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:15:29.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:15:29.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:15:29.819 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:15:29.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:15:30.389 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:15:30.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:15:30.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:15:30.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:15:30.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-08-19T15:15:30.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-08-19T15:15:30.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:15:30.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-08-19T15:15:30.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2024-08-19T15:15:30.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:15:30.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:15:30.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2024-08-19T15:15:30.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:15:30.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:15:30.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:15:31.218 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:15:31.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:15:31.753 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:15:31.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:15:31.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:15:31.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:15:32.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2024-08-19T15:15:32.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2024-08-19T15:15:32.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:15:32.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-08-19T15:15:32.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2024-08-19T15:15:32.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:15:32.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:15:32.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2024-08-19T15:15:32.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:15:32.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:15:32.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:15:32.665 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:15:32.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:15:33.143 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:15:33.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:15:33.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:15:33.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:15:33.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2024-08-19T15:15:33.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2024-08-19T15:15:33.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:15:33.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-08-19T15:15:33.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2024-08-19T15:15:33.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:15:33.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:15:33.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2024-08-19T15:15:33.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:15:33.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:15:33.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:15:33.971 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:15:33.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:15:34.339 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:15:34.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:15:34.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:15:34.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:15:34.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2024-08-19T15:15:34.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2024-08-19T15:15:34.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:15:34.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-08-19T15:15:34.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2024-08-19T15:15:34.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:15:34.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:15:34.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2024-08-19T15:15:34.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:15:34.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:15:34.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:15:35.227 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:15:35.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:15:35.612 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:15:35.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:15:35.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:15:35.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:15:35.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2024-08-19T15:15:35.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2024-08-19T15:15:36.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:15:36.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-08-19T15:15:36.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2024-08-19T15:15:36.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:15:36.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:15:36.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2024-08-19T15:15:36.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:15:36.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:15:36.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:15:36.515 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:15:36.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:15:36.993 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:15:37.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:15:37.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:15:37.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:15:37.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2024-08-19T15:15:37.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2024-08-19T15:15:37.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:15:37.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-08-19T15:15:37.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2024-08-19T15:15:37.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:15:37.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:15:37.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2024-08-19T15:15:37.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:15:37.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:15:37.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:15:37.795 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:15:37.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:15:38.323 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:15:38.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:15:38.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:15:38.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:15:38.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2024-08-19T15:15:38.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2024-08-19T15:15:38.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:15:38.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-08-19T15:15:38.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2024-08-19T15:15:38.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:15:38.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:15:38.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2024-08-19T15:15:38.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:15:38.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:15:38.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:15:39.093 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:15:39.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:15:39.469 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:15:39.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:15:39.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:15:39.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:15:39.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2024-08-19T15:15:39.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2024-08-19T15:15:40.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:15:40.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-08-19T15:15:40.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2024-08-19T15:15:40.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:15:40.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:15:40.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2024-08-19T15:15:40.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:15:40.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:15:40.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:15:40.366 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:15:40.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:15:40.743 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:15:40.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:15:40.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:15:40.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:15:40.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2024-08-19T15:15:41.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2024-08-19T15:15:41.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:15:41.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-08-19T15:15:41.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2024-08-19T15:15:41.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:15:41.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:15:41.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2024-08-19T15:15:41.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:15:41.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:15:41.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:15:41.588 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:15:41.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:15:42.032 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:15:42.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:15:42.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:15:42.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:15:42.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2024-08-19T15:15:42.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2024-08-19T15:15:42.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:15:42.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-08-19T15:15:42.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2024-08-19T15:15:42.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:15:42.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:15:42.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2024-08-19T15:15:42.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:15:42.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:15:42.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:15:42.851 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:15:42.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:15:43.354 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:15:43.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:15:43.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:15:43.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:15:43.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2024-08-19T15:15:43.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2024-08-19T15:15:43.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:15:43.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-08-19T15:15:43.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2024-08-19T15:15:43.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:15:43.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:15:43.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2024-08-19T15:15:43.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:15:43.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:15:43.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:15:44.191 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:15:44.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:15:44.896 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:15:44.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:15:44.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:15:44.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:15:45.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2024-08-19T15:15:45.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2024-08-19T15:15:45.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:15:45.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-08-19T15:15:45.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2024-08-19T15:15:45.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:15:45.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:15:45.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2024-08-19T15:15:45.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:15:45.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:15:45.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:15:45.682 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:15:45.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:15:46.210 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:15:46.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:15:46.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:15:46.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:15:46.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2024-08-19T15:15:46.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2024-08-19T15:15:46.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:15:46.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-08-19T15:15:46.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2024-08-19T15:15:46.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:15:46.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:15:46.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2024-08-19T15:15:46.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:15:46.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:15:46.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:15:47.022 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:15:47.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:15:47.524 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:15:47.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:15:47.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:15:47.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:15:47.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2024-08-19T15:15:47.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2024-08-19T15:15:48.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:15:48.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-08-19T15:15:48.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2024-08-19T15:15:48.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:15:48.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:15:48.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2024-08-19T15:15:48.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:15:48.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:15:48.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:15:48.386 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:15:48.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:15:48.754 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:15:48.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:15:48.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:15:48.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:15:48.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2024-08-19T15:15:49.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2024-08-19T15:15:49.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:15:49.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-08-19T15:15:49.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2024-08-19T15:15:49.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:15:49.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:15:49.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2024-08-19T15:15:49.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:15:49.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:15:49.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:15:49.633 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:15:49.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:15:50.035 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:15:50.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:15:50.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:15:50.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:15:50.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2024-08-19T15:15:50.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2024-08-19T15:15:50.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:15:50.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-08-19T15:15:50.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2024-08-19T15:15:50.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:15:50.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:15:50.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2024-08-19T15:15:50.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:15:50.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:15:50.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:15:50.931 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:15:50.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:15:51.408 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:15:51.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:15:51.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:15:51.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:15:51.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2024-08-19T15:15:51.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2024-08-19T15:15:51.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:15:51.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-08-19T15:15:51.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2024-08-19T15:15:51.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:15:51.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:15:51.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2024-08-19T15:15:51.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:15:51.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:15:51.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:15:52.262 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:15:52.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:15:52.655 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:15:52.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:15:52.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:15:52.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:15:52.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2024-08-19T15:15:53.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2024-08-19T15:15:53.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1015: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2024-08-19T15:15:53.231 INFO:tasks.workunit.client.0.smithi102.stderr:1025+0 records in 2024-08-19T15:15:53.231 INFO:tasks.workunit.client.0.smithi102.stderr:1025+0 records out 2024-08-19T15:15:53.231 INFO:tasks.workunit.client.0.smithi102.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00181583 s, 578 MB/s 2024-08-19T15:15:53.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2024-08-19T15:15:53.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2024-08-19T15:15:53.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-08-19T15:15:53.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-08-19T15:15:53.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-08-19T15:15:53.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-08-19T15:15:53.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:15:53.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: local pg=3.0 2024-08-19T15:15:53.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2024-08-19T15:15:53.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-08-19T15:15:53.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-08-19T15:15:53.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-08-19T15:15:53.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:15:53.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: local primary=1 2024-08-19T15:15:53.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2024-08-19T15:15:53.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T15:15:53.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T15:15:53.916 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:15:53.916 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:15:53.916 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:15:53.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.0.asok 2024-08-19T15:15:53.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-08-19T15:15:53.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.57280/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2024-08-19T15:15:53.993 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:15:53.993 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-08-19T15:15:53.993 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:15:54.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2024-08-19T15:15:54.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T15:15:54.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T15:15:54.001 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:15:54.001 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:15:54.001 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:15:54.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.1.asok 2024-08-19T15:15:54.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-08-19T15:15:54.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.57280/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2024-08-19T15:15:54.078 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:15:54.078 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-08-19T15:15:54.078 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:15:54.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-08-19T15:15:54.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-08-19T15:15:54.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:15:54.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:15:54.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:15:54.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:15:54.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:15:54.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-08-19T15:15:23.716626+0000 2024-08-19T15:15:54.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-08-19T15:15:54.601 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-08-19T15:15:54.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-08-19T15:15:23.716626+0000 last_deep_scrub_stamp 2024-08-19T15:15:54.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-08-19T15:15:54.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:15:23.716626+0000 2024-08-19T15:15:54.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-08-19T15:15:54.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:15:54.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:15:54.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:15:54.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:15:54.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:15:54.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:15:54.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:15:54.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:15:23.716626+0000 '>' 2024-08-19T15:15:23.716626+0000 2024-08-19T15:15:54.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:15:55.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:15:55.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:15:55.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:15:55.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:15:55.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:15:55.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:15:55.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:15:56.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:15:23.716626+0000 '>' 2024-08-19T15:15:23.716626+0000 2024-08-19T15:15:56.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:15:57.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:15:57.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:15:57.180 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:15:57.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:15:57.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:15:57.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:15:57.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:15:57.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:15:23.716626+0000 '>' 2024-08-19T15:15:23.716626+0000 2024-08-19T15:15:57.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:15:58.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:15:58.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:15:58.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:15:58.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:15:58.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:15:58.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:15:58.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:15:58.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:15:23.716626+0000 '>' 2024-08-19T15:15:23.716626+0000 2024-08-19T15:15:58.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:15:59.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:15:59.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:15:59.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:15:59.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:15:59.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:15:59.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:15:59.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:16:00.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:15:23.716626+0000 '>' 2024-08-19T15:15:23.716626+0000 2024-08-19T15:16:00.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:16:01.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:16:01.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:16:01.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:16:01.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:16:01.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:16:01.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:16:01.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:16:01.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:15:23.716626+0000 '>' 2024-08-19T15:15:23.716626+0000 2024-08-19T15:16:01.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:16:02.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:16:02.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:16:02.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:16:02.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:16:02.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:16:02.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:16:02.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:16:02.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:15:55.624899+0000 '>' 2024-08-19T15:15:23.716626+0000 2024-08-19T15:16:02.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:16:02.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: seq 1 19 2024-08-19T15:16:02.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:16:02.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-08-19T15:16:02.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 1 % 2 2024-08-19T15:16:02.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-08-19T15:16:02.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-08-19T15:16:02.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1038: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2024-08-19T15:16:02.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1039: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2024-08-19T15:16:02.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:16:02.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:16:02.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:16:02.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:16:02.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:16:02.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:16:02.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:16:02.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:16:02.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:16:02.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:16:02.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:16:02.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:16:02.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:16:02.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:16:02.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:16:02.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:16:02.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:16:02.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:16:02.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:16:02.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:16:02.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:16:02.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:16:02.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:16:02.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:16:04.223 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2024-08-19T15:16:04.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:16:04.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:16:04.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:16:04.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:16:04.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:16:04.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:16:04.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:16:04.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:16:04.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:16:04.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:16:04.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:16:04.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:16:04.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:16:04.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:16:04.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:16:04.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:16:04.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:16:04.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:16:04.770 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:16:04.770 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:16:04.770 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:16:04.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:16:04.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:16:04.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:16:04.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:16:04.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:16:04.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:16:04.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:16:04.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:16:04.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:16:04.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:16:04.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:16:04.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:16:04.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:16:04.771 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:16:04.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:16:04.772 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:16:04.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:16:04.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:16:04.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:16:04.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:16:04.798 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:04.797+0000 7fc8bde1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:16:04.801 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:04.800+0000 7fc8bde1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:16:04.802 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:04.801+0000 7fc8bde1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:16:05.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:16:05.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:16:05.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:16:05.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:16:05.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:16:05.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:05.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:16:05.051 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:16:05.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:05.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:16:05.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:16:05.332 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:05.330+0000 7fc8bde1b640 -1 Falling back to public interface 2024-08-19T15:16:06.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:16:06.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:06.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:16:06.318 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:16:06.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:06.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:16:06.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:16:07.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:16:07.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:07.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:16:07.589 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:16:07.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:07.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:16:07.676 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:07.674+0000 7fc8bde1b640 -1 osd.1 61 log_to_monitors true 2024-08-19T15:16:07.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:16:08.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:16:08.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:08.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:16:08.876 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:16:08.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:08.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:16:09.236 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 65 up_thru 65 down_at 62 last_clean_interval [10,61) [v2:127.0.0.1:6810/3556416586,v1:127.0.0.1:6811/3556416586] [v2:127.0.0.1:6812/3556416586,v1:127.0.0.1:6813/3556416586] exists,up 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:16:09.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:16:09.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:16:09.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:16:09.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:16:09.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:16:09.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:16:09.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:16:09.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:16:09.237 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:16:09.237 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:16:09.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:16:09.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:16:09.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:16:09.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:16:09.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:16:09.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:16:09.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:16:09.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:16:09.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:16:09.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:16:09.630 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:16:09.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:16:09.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:16:09.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:16:09.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-08-19T15:16:09.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-08-19T15:16:09.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-08-19T15:16:09.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:16:09.767 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:16:09.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874242 2024-08-19T15:16:09.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874242 2024-08-19T15:16:09.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-279172874242' 2024-08-19T15:16:09.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:16:09.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-08-19T15:16:09.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:16:09.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:16:09.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-08-19T15:16:09.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:16:09.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-08-19T15:16:09.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-08-19T15:16:09.873 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836504 2024-08-19T15:16:09.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:16:10.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-08-19T15:16:10.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:16:10.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-279172874242 2024-08-19T15:16:10.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:16:10.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:16:10.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-279172874242 2024-08-19T15:16:10.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:16:10.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874242 2024-08-19T15:16:10.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 279172874242' 2024-08-19T15:16:10.156 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 279172874242 2024-08-19T15:16:10.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:16:10.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 279172874242 2024-08-19T15:16:10.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:16:11.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:16:11.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:16:11.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 279172874242 2024-08-19T15:16:11.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:16:12.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:16:12.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:16:13.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874242 -lt 279172874242 2024-08-19T15:16:13.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:16:13.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:16:13.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:16:13.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:16:13.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:16:13.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:16:13.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:16:13.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:16:13.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:16:13.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:16:13.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:16:13.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:16:13.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:16:13.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:16:13.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:16:13.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:16:13.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:16:13.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:16:13.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:16:13.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-08-19T15:16:13.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 2 % 2 2024-08-19T15:16:13.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-08-19T15:16:13.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-08-19T15:16:13.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1045: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2024-08-19T15:16:13.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: echo UVWXYZ 2024-08-19T15:16:13.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:16:13.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:16:13.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:16:13.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:16:13.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:16:13.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:16:13.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:16:13.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:16:13.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:16:13.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:16:13.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:16:13.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:16:13.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:16:13.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:16:13.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:16:13.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:16:14.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:16:14.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:16:14.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:16:14.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:16:14.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:16:14.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:16:14.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:16:14.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:16:15.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:16:15.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:16:15.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:16:15.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:16:15.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:16:15.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:16:15.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:16:15.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:16:15.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:16:15.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:16:15.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:16:15.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:16:15.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:16:15.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:16:15.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:16:15.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:16:15.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:16:15.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:16:15.949 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:16:15.949 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:16:15.949 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:16:15.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:16:15.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:16:15.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:16:15.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:16:15.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:16:15.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:16:15.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:16:15.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:16:15.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:16:15.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:16:15.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:16:15.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:16:15.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:16:15.951 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:16:15.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:16:15.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:16:15.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:16:15.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:16:15.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:16:15.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:16:15.978 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:15.977+0000 7f4e19d2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:16:15.989 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:15.988+0000 7f4e19d2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:16:15.990 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:15.989+0000 7f4e19d2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:16:16.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:16:16.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:16:16.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:16:16.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:16:16.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:16:16.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:16.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:16:16.228 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:16:16.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:16.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:16:16.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:16:16.508 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:16.507+0000 7f4e19d2c640 -1 Falling back to public interface 2024-08-19T15:16:17.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:16:17.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:17.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:16:17.506 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:16:17.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:17.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:16:17.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:16:18.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:16:18.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:18.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:16:18.782 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:16:18.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:18.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:16:18.821 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:18.819+0000 7f4e19d2c640 -1 osd.0 67 log_to_monitors true 2024-08-19T15:16:19.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:16:20.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:16:20.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:20.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:16:20.154 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:16:20.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:20.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:16:20.456 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 71 up_thru 71 down_at 68 last_clean_interval [5,67) [v2:127.0.0.1:6802/1590525281,v1:127.0.0.1:6803/1590525281] [v2:127.0.0.1:6804/1590525281,v1:127.0.0.1:6805/1590525281] exists,up a9a4af16-d461-4c2f-ab0d-b99df37a8a3e 2024-08-19T15:16:20.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:16:20.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:16:20.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:16:20.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:16:20.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:16:20.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:16:20.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:16:20.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:16:20.457 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:16:20.457 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:16:20.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:16:20.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:16:20.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:16:20.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:16:20.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:16:20.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:16:20.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:16:20.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:16:20.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:16:20.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:16:20.815 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:16:20.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:16:20.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:16:20.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:16:20.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678018 2024-08-19T15:16:20.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678018 2024-08-19T15:16:20.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-304942678018' 2024-08-19T15:16:20.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:16:20.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:16:21.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874245 2024-08-19T15:16:21.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874245 2024-08-19T15:16:21.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-304942678018 1-279172874245' 2024-08-19T15:16:21.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:16:21.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-304942678018 2024-08-19T15:16:21.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:16:21.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:16:21.024 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-304942678018 2024-08-19T15:16:21.024 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:16:21.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678018 2024-08-19T15:16:21.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 304942678018' 2024-08-19T15:16:21.025 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 304942678018 2024-08-19T15:16:21.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:16:21.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 304942678018 2024-08-19T15:16:21.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:16:22.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:16:22.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:16:22.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678018 -lt 304942678018 2024-08-19T15:16:22.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:16:22.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-279172874245 2024-08-19T15:16:22.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:16:22.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:16:22.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-279172874245 2024-08-19T15:16:22.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:16:22.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874245 2024-08-19T15:16:22.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 279172874245' 2024-08-19T15:16:22.571 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 279172874245 2024-08-19T15:16:22.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:16:22.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874245 -lt 279172874245 2024-08-19T15:16:22.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:16:22.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:16:22.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:16:23.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:16:23.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:16:23.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:16:23.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:16:23.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:16:23.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:16:23.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:16:23.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:16:23.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:16:23.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:16:23.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:16:23.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:16:23.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:16:23.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:16:23.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:16:23.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:16:23.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-08-19T15:16:23.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 3 % 2 2024-08-19T15:16:23.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-08-19T15:16:23.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-08-19T15:16:23.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2024-08-19T15:16:23.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:16:23.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:16:23.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:16:23.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:16:23.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2024-08-19T15:16:23.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:16:23.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:16:23.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:16:23.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:16:23.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:16:23.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:16:23.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:16:23.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:16:23.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:16:23.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:16:24.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:16:24.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2024-08-19T15:16:24.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:16:24.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:16:24.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:16:24.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:16:24.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:16:24.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2024-08-19T15:16:25.133 INFO:tasks.workunit.client.0.smithi102.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2024-08-19T15:16:25.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:16:25.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:16:25.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:16:25.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:16:25.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:16:25.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:16:25.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:16:25.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:16:25.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:16:25.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:16:25.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:16:25.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:16:25.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:16:25.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:16:25.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:16:25.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:16:25.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:16:25.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:16:25.651 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:16:25.651 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:16:25.651 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:16:25.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:16:25.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:16:25.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:16:25.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:16:25.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:16:25.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:16:25.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:16:25.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:16:25.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:16:25.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:16:25.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:16:25.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:16:25.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:16:25.653 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:16:25.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:16:25.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:16:25.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:16:25.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:16:25.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:16:25.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:16:25.681 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:25.679+0000 7f5f8ad21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:16:25.683 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:25.682+0000 7f5f8ad21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:16:25.684 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:25.683+0000 7f5f8ad21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:16:25.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:16:25.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:16:25.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:16:25.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:16:25.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:16:25.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:25.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:16:25.931 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:16:25.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:25.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:16:26.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:16:26.203 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:26.202+0000 7f5f8ad21640 -1 Falling back to public interface 2024-08-19T15:16:27.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:16:27.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:27.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:16:27.197 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:16:27.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:27.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:16:27.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:16:28.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:16:28.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:28.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:16:28.469 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:16:28.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:28.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:16:28.549 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:28.548+0000 7f5f8ad21640 -1 osd.1 73 log_to_monitors true 2024-08-19T15:16:28.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:16:29.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:16:29.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:29.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:16:29.852 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:16:29.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:29.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:16:30.155 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 77 up_thru 77 down_at 74 last_clean_interval [65,73) [v2:127.0.0.1:6810/1474083987,v1:127.0.0.1:6811/1474083987] [v2:127.0.0.1:6812/1474083987,v1:127.0.0.1:6813/1474083987] exists,up 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:16:30.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:16:30.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:16:30.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:16:30.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:16:30.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:16:30.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:16:30.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:16:30.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:16:30.156 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:16:30.156 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:16:30.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:16:30.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:16:30.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:16:30.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:16:30.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:16:30.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:16:30.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:16:30.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:16:30.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:16:30.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:16:30.514 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:16:30.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:16:30.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:16:30.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:16:30.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678021 2024-08-19T15:16:30.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678021 2024-08-19T15:16:30.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-304942678021' 2024-08-19T15:16:30.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:16:30.616 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:16:30.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-08-19T15:16:30.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-08-19T15:16:30.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-304942678021 1-330712481794' 2024-08-19T15:16:30.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:16:30.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-304942678021 2024-08-19T15:16:30.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:16:30.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:16:30.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-304942678021 2024-08-19T15:16:30.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:16:30.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678021 2024-08-19T15:16:30.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 304942678021' 2024-08-19T15:16:30.721 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 304942678021 2024-08-19T15:16:30.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:16:30.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678020 -lt 304942678021 2024-08-19T15:16:30.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:16:31.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:16:31.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:16:32.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678021 -lt 304942678021 2024-08-19T15:16:32.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:16:32.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481794 2024-08-19T15:16:32.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:16:32.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:16:32.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481794 2024-08-19T15:16:32.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:16:32.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-08-19T15:16:32.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2024-08-19T15:16:32.261 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 330712481794 2024-08-19T15:16:32.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:16:32.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481794 -lt 330712481794 2024-08-19T15:16:32.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:16:32.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:16:32.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:16:32.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:16:32.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:16:32.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:16:32.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:16:32.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:16:32.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:16:32.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:16:32.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:16:33.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:16:33.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:16:33.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:16:33.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:16:33.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:16:33.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:16:33.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:16:33.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:16:33.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-08-19T15:16:33.484 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 4 % 2 2024-08-19T15:16:33.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-08-19T15:16:33.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-08-19T15:16:33.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-08-19T15:16:33.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:16:33.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:16:33.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:16:33.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:16:33.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-08-19T15:16:33.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:16:33.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:16:33.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:16:33.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:16:33.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:16:33.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:16:33.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:16:33.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:16:33.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:16:33.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:16:33.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:16:33.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-08-19T15:16:33.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:16:33.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:16:33.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:16:33.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:16:33.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:16:33.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-08-19T15:16:35.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:16:35.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:16:35.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:16:35.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:16:35.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:16:35.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:16:35.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:16:35.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:16:35.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:16:35.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:16:35.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:16:35.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:16:35.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:16:35.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:16:35.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:16:35.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:16:35.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:16:35.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:16:35.516 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:16:35.516 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:16:35.516 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:16:35.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:16:35.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:16:35.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:16:35.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:16:35.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:16:35.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:16:35.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:16:35.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:16:35.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:16:35.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:16:35.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:16:35.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:16:35.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:16:35.519 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:16:35.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:16:35.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:16:35.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:16:35.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:16:35.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:16:35.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:16:35.547 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:35.546+0000 7f879ca0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:16:35.549 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:35.548+0000 7f879ca0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:16:35.550 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:35.550+0000 7f879ca0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:16:35.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:16:35.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:16:35.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:16:35.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:16:35.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:16:35.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:35.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:16:35.796 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:16:35.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:35.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:16:36.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:16:36.084 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:36.083+0000 7f879ca0e640 -1 Falling back to public interface 2024-08-19T15:16:37.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:16:37.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:37.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:16:37.063 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:16:37.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:37.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:16:37.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:16:38.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:16:38.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:38.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:16:38.335 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:16:38.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:38.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:16:38.465 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:38.464+0000 7f879ca0e640 -1 osd.0 78 log_to_monitors true 2024-08-19T15:16:38.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:16:39.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:16:39.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:39.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:16:39.611 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:16:39.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:39.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:16:39.950 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [71,78) [v2:127.0.0.1:6802/2842598710,v1:127.0.0.1:6803/2842598710] [v2:127.0.0.1:6804/2842598710,v1:127.0.0.1:6805/2842598710] exists,up a9a4af16-d461-4c2f-ab0d-b99df37a8a3e 2024-08-19T15:16:39.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:16:39.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:16:39.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:16:39.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:16:39.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:16:39.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:16:39.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:16:39.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:16:39.951 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:16:39.951 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:16:39.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:16:39.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:16:39.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:16:40.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:16:40.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:16:40.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:16:40.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:16:40.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:16:40.044 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:16:40.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:16:40.311 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:16:40.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:16:40.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:16:40.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:16:40.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-08-19T15:16:40.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-08-19T15:16:40.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318274' 2024-08-19T15:16:40.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:16:40.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:16:40.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481797 2024-08-19T15:16:40.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481797 2024-08-19T15:16:40.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318274 1-330712481797' 2024-08-19T15:16:40.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:16:40.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-352187318274 2024-08-19T15:16:40.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:16:40.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:16:40.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-352187318274 2024-08-19T15:16:40.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:16:40.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-08-19T15:16:40.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 352187318274' 2024-08-19T15:16:40.520 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 352187318274 2024-08-19T15:16:40.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:16:40.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 352187318274 2024-08-19T15:16:40.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:16:41.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:16:41.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:16:42.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318274 -lt 352187318274 2024-08-19T15:16:42.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:16:42.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481797 2024-08-19T15:16:42.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:16:42.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:16:42.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481797 2024-08-19T15:16:42.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:16:42.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481797 2024-08-19T15:16:42.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481797' 2024-08-19T15:16:42.075 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 330712481797 2024-08-19T15:16:42.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:16:42.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481797 -lt 330712481797 2024-08-19T15:16:42.373 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:16:42.373 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:16:42.373 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:16:42.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:16:42.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:16:42.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:16:42.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:16:42.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:16:42.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:16:42.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:16:42.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:16:42.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:16:42.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:16:42.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:16:42.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:16:43.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:16:43.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:16:43.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:16:43.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:16:43.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-08-19T15:16:43.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 5 % 2 2024-08-19T15:16:43.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-08-19T15:16:43.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-08-19T15:16:43.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-08-19T15:16:43.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:16:43.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:16:43.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:16:43.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:16:43.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-08-19T15:16:43.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:16:43.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:16:43.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:16:43.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:16:43.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:16:43.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:16:43.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:16:43.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:16:43.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:16:43.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:16:43.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:16:43.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-08-19T15:16:43.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:16:43.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:16:43.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:16:43.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:16:43.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:16:43.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2024-08-19T15:16:45.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:16:45.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:16:45.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:16:45.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:16:45.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:16:45.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:16:45.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:16:45.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:16:45.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:16:45.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:16:45.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:16:45.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:16:45.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:16:45.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:16:45.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:16:45.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:16:45.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:16:45.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:16:45.422 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:16:45.422 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:16:45.422 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:16:45.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:16:45.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:16:45.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:16:45.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:16:45.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:16:45.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:16:45.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:16:45.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:16:45.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:16:45.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:16:45.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:16:45.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:16:45.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:16:45.423 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:16:45.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:16:45.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:16:45.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:16:45.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:16:45.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:16:45.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:16:45.451 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:45.449+0000 7f560d13b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:16:45.453 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:45.452+0000 7f560d13b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:16:45.454 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:45.453+0000 7f560d13b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:16:45.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:16:45.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:16:45.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:16:45.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:16:45.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:16:45.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:45.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:16:45.706 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:16:45.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:45.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:16:45.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:16:45.973 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:45.972+0000 7f560d13b640 -1 Falling back to public interface 2024-08-19T15:16:46.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:16:46.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:46.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:16:46.975 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:16:46.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:46.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:16:47.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:16:48.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:16:48.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:48.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:16:48.251 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:16:48.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:48.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:16:48.325 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:48.324+0000 7f560d13b640 -1 osd.1 84 log_to_monitors true 2024-08-19T15:16:48.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:16:49.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:16:49.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:49.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:16:49.624 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:16:49.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:49.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:16:49.894 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 88 up_thru 88 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6810/3321480215,v1:127.0.0.1:6811/3321480215] [v2:127.0.0.1:6812/3321480215,v1:127.0.0.1:6813/3321480215] exists,up 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:16:49.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:16:49.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:16:49.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:16:49.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:16:49.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:16:49.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:16:49.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:16:49.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:16:49.895 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:16:49.896 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:16:49.896 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:16:49.896 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:16:49.896 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:16:49.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:16:49.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:16:49.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:16:49.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:16:49.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:16:49.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:16:50.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:16:50.253 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:16:50.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:16:50.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:16:50.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:16:50.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318277 2024-08-19T15:16:50.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318277 2024-08-19T15:16:50.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318277' 2024-08-19T15:16:50.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:16:50.356 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:16:50.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-08-19T15:16:50.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-08-19T15:16:50.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318277 1-377957122050' 2024-08-19T15:16:50.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:16:50.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-352187318277 2024-08-19T15:16:50.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:16:50.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:16:50.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-352187318277 2024-08-19T15:16:50.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:16:50.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318277 2024-08-19T15:16:50.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 352187318277' 2024-08-19T15:16:50.462 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 352187318277 2024-08-19T15:16:50.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:16:50.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318276 -lt 352187318277 2024-08-19T15:16:50.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:16:51.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:16:51.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:16:52.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318277 -lt 352187318277 2024-08-19T15:16:52.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:16:52.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-377957122050 2024-08-19T15:16:52.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:16:52.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:16:52.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-377957122050 2024-08-19T15:16:52.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:16:52.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-08-19T15:16:52.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 377957122050' 2024-08-19T15:16:52.034 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 377957122050 2024-08-19T15:16:52.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:16:52.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122050 -lt 377957122050 2024-08-19T15:16:52.308 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:16:52.308 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:16:52.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:16:52.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:16:52.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:16:52.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:16:52.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:16:52.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:16:52.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:16:52.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:16:52.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:16:52.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:16:52.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:16:52.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:16:52.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:16:53.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:16:53.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:16:53.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:16:53.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:16:53.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-08-19T15:16:53.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 6 % 2 2024-08-19T15:16:53.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-08-19T15:16:53.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-08-19T15:16:53.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: echo extra 2024-08-19T15:16:53.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1068: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-08-19T15:16:53.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:16:53.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:16:53.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:16:53.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:16:53.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-08-19T15:16:53.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:16:53.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:16:53.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:16:53.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:16:53.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:16:53.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:16:53.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:16:53.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:16:53.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:16:53.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:16:53.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:16:53.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-08-19T15:16:53.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:16:53.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:16:53.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:16:53.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:16:53.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:16:53.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-08-19T15:16:55.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:16:55.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:16:55.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:16:55.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:16:55.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:16:55.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:16:55.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:16:55.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:16:55.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:16:55.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:16:55.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:16:55.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:16:55.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:16:55.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:16:55.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:16:55.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:16:55.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:16:55.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:16:55.239 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:16:55.240 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:16:55.240 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:16:55.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:16:55.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:16:55.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:16:55.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:16:55.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:16:55.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:16:55.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:16:55.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:16:55.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:16:55.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:16:55.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:16:55.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:16:55.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:16:55.242 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:16:55.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:16:55.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:16:55.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:16:55.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:16:55.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:16:55.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:16:55.269 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:55.267+0000 7f26187ab640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:16:55.272 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:55.270+0000 7f26187ab640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:16:55.273 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:55.271+0000 7f26187ab640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:16:55.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:16:55.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:16:55.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:16:55.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:16:55.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:16:55.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:55.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:16:55.520 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:16:55.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:55.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:16:55.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:16:55.799 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:55.797+0000 7f26187ab640 -1 Falling back to public interface 2024-08-19T15:16:56.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:16:56.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:56.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:16:56.788 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:16:56.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:56.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:16:57.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:16:58.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:16:58.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:58.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:16:58.065 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:16:58.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:58.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:16:58.252 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:16:58.250+0000 7f26187ab640 -1 osd.0 89 log_to_monitors true 2024-08-19T15:16:58.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:16:59.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:16:59.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:16:59.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:16:59.373 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:16:59.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:16:59.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:16:59.665 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 93 up_thru 93 down_at 90 last_clean_interval [82,89) [v2:127.0.0.1:6802/286169626,v1:127.0.0.1:6803/286169626] [v2:127.0.0.1:6804/286169626,v1:127.0.0.1:6805/286169626] exists,up a9a4af16-d461-4c2f-ab0d-b99df37a8a3e 2024-08-19T15:16:59.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:16:59.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:16:59.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:16:59.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:16:59.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:16:59.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:16:59.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:16:59.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:16:59.667 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:16:59.667 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:16:59.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:16:59.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:16:59.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:16:59.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:16:59.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:16:59.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:16:59.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:16:59.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:16:59.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:17:00.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:17:00.079 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:17:00.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:17:00.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:17:00.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:17:00.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958530 2024-08-19T15:17:00.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958530 2024-08-19T15:17:00.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958530' 2024-08-19T15:17:00.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:17:00.185 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:17:00.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122053 2024-08-19T15:17:00.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122053 2024-08-19T15:17:00.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958530 1-377957122053' 2024-08-19T15:17:00.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:17:00.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-399431958530 2024-08-19T15:17:00.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:17:00.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:17:00.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-399431958530 2024-08-19T15:17:00.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:17:00.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958530 2024-08-19T15:17:00.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 399431958530' 2024-08-19T15:17:00.290 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 399431958530 2024-08-19T15:17:00.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:17:00.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 399431958530 2024-08-19T15:17:00.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:17:01.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:17:01.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:17:01.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 399431958530 2024-08-19T15:17:01.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:17:02.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:17:02.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:17:03.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958530 -lt 399431958530 2024-08-19T15:17:03.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:17:03.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-377957122053 2024-08-19T15:17:03.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:17:03.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:17:03.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-377957122053 2024-08-19T15:17:03.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:17:03.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122053 2024-08-19T15:17:03.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 377957122053' 2024-08-19T15:17:03.109 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 377957122053 2024-08-19T15:17:03.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:17:03.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122053 -lt 377957122053 2024-08-19T15:17:03.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:17:03.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:17:03.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:17:03.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:17:03.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:17:03.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:17:03.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:17:03.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:17:03.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:17:03.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:17:03.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:17:03.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:17:03.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:17:03.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:17:03.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:17:04.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:17:04.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:17:04.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:17:04.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2024-08-19T15:17:04.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:17:04.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-08-19T15:17:04.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 7 % 2 2024-08-19T15:17:04.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-08-19T15:17:04.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-08-19T15:17:04.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: echo -n newheader 2024-08-19T15:17:04.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-08-19T15:17:04.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:17:04.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:17:04.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:17:04.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:17:04.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-08-19T15:17:04.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:17:04.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:17:04.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:17:04.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:17:04.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:17:04.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:17:04.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:17:04.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:17:04.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:17:04.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:17:04.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:17:04.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-08-19T15:17:04.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:17:04.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:17:04.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:17:04.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:17:04.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:17:04.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-08-19T15:17:06.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:17:06.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:17:06.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:17:06.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:17:06.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:17:06.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:17:06.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:17:06.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:17:06.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:17:06.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:17:06.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:17:06.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:17:06.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:17:06.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:17:06.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:17:06.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:17:06.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:17:06.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:17:06.115 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:17:06.115 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:17:06.115 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:17:06.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:17:06.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:17:06.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:17:06.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:17:06.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:17:06.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:17:06.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:17:06.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:17:06.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:17:06.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:17:06.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:17:06.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:17:06.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:17:06.117 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:17:06.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:17:06.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:17:06.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:17:06.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:17:06.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:17:06.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:17:06.142 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:06.140+0000 7f8dadb42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:17:06.144 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:06.143+0000 7f8dadb42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:17:06.145 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:06.144+0000 7f8dadb42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:17:06.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:17:06.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:17:06.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:17:06.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:17:06.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:17:06.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:06.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:17:06.389 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:17:06.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:06.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:17:06.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:17:06.671 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:06.670+0000 7f8dadb42640 -1 Falling back to public interface 2024-08-19T15:17:07.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:17:07.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:07.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:17:07.655 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:17:07.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:07.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:17:07.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:17:08.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:17:08.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:08.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:17:08.928 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:17:08.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:08.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:17:09.047 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:09.046+0000 7f8dadb42640 -1 osd.1 95 log_to_monitors true 2024-08-19T15:17:09.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:17:10.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:17:10.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:10.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:17:10.220 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:17:10.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:10.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:17:10.561 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 99 up_thru 99 down_at 96 last_clean_interval [88,95) [v2:127.0.0.1:6810/205352386,v1:127.0.0.1:6811/205352386] [v2:127.0.0.1:6812/205352386,v1:127.0.0.1:6813/205352386] exists,up 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:17:10.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:17:10.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:17:10.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:17:10.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:17:10.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:17:10.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:17:10.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:17:10.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:17:10.563 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:17:10.563 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:17:10.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:17:10.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:17:10.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:17:10.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:17:10.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:17:10.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:17:10.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:17:10.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:17:10.656 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:17:10.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:17:10.921 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:17:10.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:17:10.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:17:10.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:17:11.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958533 2024-08-19T15:17:11.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958533 2024-08-19T15:17:11.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958533' 2024-08-19T15:17:11.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:17:11.024 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:17:11.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762306 2024-08-19T15:17:11.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762306 2024-08-19T15:17:11.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958533 1-425201762306' 2024-08-19T15:17:11.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:17:11.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-399431958533 2024-08-19T15:17:11.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:17:11.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:17:11.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-399431958533 2024-08-19T15:17:11.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:17:11.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958533 2024-08-19T15:17:11.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 399431958533' 2024-08-19T15:17:11.131 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 399431958533 2024-08-19T15:17:11.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:17:11.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958532 -lt 399431958533 2024-08-19T15:17:11.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:17:12.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:17:12.398 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:17:12.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958533 -lt 399431958533 2024-08-19T15:17:12.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:17:12.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-425201762306 2024-08-19T15:17:12.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:17:12.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:17:12.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-425201762306 2024-08-19T15:17:12.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:17:12.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762306 2024-08-19T15:17:12.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 425201762306' 2024-08-19T15:17:12.676 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 425201762306 2024-08-19T15:17:12.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:17:12.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762306 -lt 425201762306 2024-08-19T15:17:12.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:17:12.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:17:12.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:17:13.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:17:13.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:17:13.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:17:13.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:17:13.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:17:13.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:17:13.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:17:13.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:17:13.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:17:13.577 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:17:13.577 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:17:13.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:17:13.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:17:13.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:17:13.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:17:13.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1076: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2024-08-19T15:17:13.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:17:13.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-08-19T15:17:13.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 8 % 2 2024-08-19T15:17:13.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-08-19T15:17:13.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-08-19T15:17:13.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1080: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2024-08-19T15:17:14.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1081: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2024-08-19T15:17:14.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-08-19T15:17:14.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1085: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-08-19T15:17:14.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:17:14.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:17:14.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:17:14.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:17:14.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-08-19T15:17:14.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:17:14.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:17:14.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:17:14.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:17:14.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:17:14.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:17:14.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:17:14.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:17:14.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:17:14.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:17:14.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:17:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-08-19T15:17:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:17:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:17:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:17:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:17:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:17:14.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-08-19T15:17:15.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:17:15.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:17:15.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:17:15.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:17:15.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:17:15.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:17:15.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:17:15.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:17:15.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:17:15.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:17:15.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:17:15.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:17:15.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:17:15.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:17:15.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:17:15.858 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:17:15.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:17:15.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:17:15.859 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:17:15.859 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:17:15.859 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:17:15.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:17:15.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:17:15.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:17:15.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:17:15.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:17:15.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:17:15.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:17:15.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:17:15.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:17:15.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:17:15.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:17:15.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:17:15.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:17:15.862 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:17:15.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:17:15.862 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:17:15.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:17:15.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:17:15.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:17:15.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:17:15.887 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:15.886+0000 7f5b0fa24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:17:15.890 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:15.889+0000 7f5b0fa24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:17:15.891 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:15.890+0000 7f5b0fa24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:17:16.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:17:16.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:17:16.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:17:16.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:17:16.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:17:16.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:16.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:17:16.139 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:17:16.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:16.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:17:16.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:17:16.408 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:16.407+0000 7f5b0fa24640 -1 Falling back to public interface 2024-08-19T15:17:17.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:17:17.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:17.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:17:17.408 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:17:17.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:17.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:17:17.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:17:18.607 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:18.606+0000 7f5b0fa24640 -1 osd.0 101 log_to_monitors true 2024-08-19T15:17:18.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:17:18.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:18.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:17:18.684 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:17:18.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:18.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:17:18.984 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 105 up_thru 100 down_at 102 last_clean_interval [93,101) [v2:127.0.0.1:6802/2985283044,v1:127.0.0.1:6803/2985283044] [v2:127.0.0.1:6804/2985283044,v1:127.0.0.1:6805/2985283044] exists,up a9a4af16-d461-4c2f-ab0d-b99df37a8a3e 2024-08-19T15:17:18.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:17:18.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:17:18.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:17:18.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:17:18.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:17:18.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:17:18.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:17:18.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:17:18.986 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:17:18.986 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:17:18.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:17:18.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:17:18.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:17:19.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:17:19.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:17:19.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:17:19.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:17:19.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:17:19.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:17:19.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:17:19.494 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:17:19.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:17:19.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:17:19.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:17:19.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566082 2024-08-19T15:17:19.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566082 2024-08-19T15:17:19.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566082' 2024-08-19T15:17:19.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:17:19.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:17:19.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762309 2024-08-19T15:17:19.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762309 2024-08-19T15:17:19.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566082 1-425201762309' 2024-08-19T15:17:19.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:17:19.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566082 2024-08-19T15:17:19.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:17:19.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:17:19.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566082 2024-08-19T15:17:19.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:17:19.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566082 2024-08-19T15:17:19.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566082' 2024-08-19T15:17:19.747 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 450971566082 2024-08-19T15:17:19.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:17:20.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566082 -lt 450971566082 2024-08-19T15:17:20.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:17:20.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-425201762309 2024-08-19T15:17:20.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:17:20.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:17:20.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-425201762309 2024-08-19T15:17:20.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:17:20.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762309 2024-08-19T15:17:20.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 425201762309' 2024-08-19T15:17:20.032 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 425201762309 2024-08-19T15:17:20.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:17:20.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762309 -lt 425201762309 2024-08-19T15:17:20.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:17:20.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:17:20.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:17:20.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:17:20.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:17:20.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:17:20.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:17:20.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:17:20.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:17:20.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:17:20.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:17:20.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-08-19T15:17:20.889 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:17:20.889 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:17:20.889 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:17:21.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-08-19T15:17:21.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-08-19T15:17:21.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T15:17:21.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-08-19T15:17:21.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T15:17:21.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T15:17:21.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T15:17:21.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:17:21.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:17:21.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:17:21.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:17:21.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:17:21.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:17:21.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:17:21.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-08-19T15:17:21.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:17:21.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:17:21.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:17:21.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-08-19T15:17:21.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-08-19T15:17:21.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T15:17:21.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T15:17:21.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T15:17:21.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T15:17:21.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T15:17:21.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T15:17:21.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T15:17:22.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T15:17:22.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T15:17:22.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-19T15:17:22.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T15:17:22.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-19T15:17:22.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T15:17:22.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:17:22.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:17:22.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:17:22.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:17:22.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:17:22.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:17:22.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:17:22.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-08-19T15:17:22.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:17:22.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:17:22.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:17:23.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-08-19T15:17:23.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-08-19T15:17:23.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T15:17:23.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T15:17:23.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T15:17:23.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T15:17:23.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T15:17:23.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T15:17:23.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T15:17:23.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T15:17:23.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T15:17:23.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-08-19T15:17:23.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T15:17:23.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-19T15:17:23.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T15:17:23.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:17:23.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:17:23.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:17:23.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:17:23.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:17:23.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:17:23.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:17:24.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:17:24.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:17:24.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:17:24.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:17:24.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:17:24.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:17:24.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:17:24.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-08-19T15:17:24.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:17:24.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:17:24.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:17:24.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:17:24.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-08-19T15:17:24.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:17:24.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:17:24.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:17:24.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:17:24.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:17:24.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:17:24.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:17:24.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:17:24.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:17:24.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:17:24.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:17:24.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-08-19T15:17:24.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:17:24.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:17:24.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:17:24.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:17:24.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:17:24.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2024-08-19T15:17:26.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:17:26.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:17:26.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:17:26.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:17:26.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:17:26.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:17:26.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:17:26.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:17:26.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:17:26.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:17:26.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:17:26.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:17:26.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:17:26.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:17:26.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:17:26.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:17:26.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:17:26.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:17:26.268 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:17:26.268 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:17:26.268 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:17:26.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:17:26.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:17:26.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:17:26.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:17:26.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:17:26.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:17:26.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:17:26.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:17:26.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:17:26.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:17:26.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:17:26.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:17:26.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:17:26.269 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:17:26.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:17:26.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:17:26.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:17:26.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:17:26.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:17:26.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:17:26.296 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:26.295+0000 7f438300f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:17:26.298 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:26.297+0000 7f438300f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:17:26.299 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:26.298+0000 7f438300f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:17:26.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:17:26.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:17:26.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:17:26.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:17:26.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:17:26.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:26.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:17:26.545 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:17:26.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:26.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:17:26.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:17:26.825 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:26.824+0000 7f438300f640 -1 Falling back to public interface 2024-08-19T15:17:27.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:17:27.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:27.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:17:27.809 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:17:27.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:27.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:17:28.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:17:29.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:17:29.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:29.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:17:29.088 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:17:29.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:29.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:17:29.141 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:29.140+0000 7f438300f640 -1 osd.1 106 log_to_monitors true 2024-08-19T15:17:29.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:17:30.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:17:30.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:30.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:17:30.365 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:17:30.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:30.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:17:30.685 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 110 up_thru 110 down_at 107 last_clean_interval [99,106) [v2:127.0.0.1:6810/3990598445,v1:127.0.0.1:6811/3990598445] [v2:127.0.0.1:6812/3990598445,v1:127.0.0.1:6813/3990598445] exists,up 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:17:30.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:17:30.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:17:30.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:17:30.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:17:30.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:17:30.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:17:30.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:17:30.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:17:30.686 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:17:30.686 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:17:30.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:17:30.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:17:30.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:17:30.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:17:30.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:17:30.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:17:30.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:17:30.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:17:30.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:17:31.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:17:31.045 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:17:31.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:17:31.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:17:31.045 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:17:31.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566085 2024-08-19T15:17:31.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566085 2024-08-19T15:17:31.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566085' 2024-08-19T15:17:31.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:17:31.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:17:31.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402562 2024-08-19T15:17:31.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402562 2024-08-19T15:17:31.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566085 1-472446402562' 2024-08-19T15:17:31.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:17:31.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566085 2024-08-19T15:17:31.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:17:31.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:17:31.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566085 2024-08-19T15:17:31.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:17:31.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566085 2024-08-19T15:17:31.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566085' 2024-08-19T15:17:31.253 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 450971566085 2024-08-19T15:17:31.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:17:31.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566084 -lt 450971566085 2024-08-19T15:17:31.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:17:32.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:17:32.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:17:32.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566085 -lt 450971566085 2024-08-19T15:17:32.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:17:32.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-472446402562 2024-08-19T15:17:32.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:17:32.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:17:32.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-472446402562 2024-08-19T15:17:32.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:17:32.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402562 2024-08-19T15:17:32.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 472446402562' 2024-08-19T15:17:32.792 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 472446402562 2024-08-19T15:17:32.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:17:33.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402562 -lt 472446402562 2024-08-19T15:17:33.058 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:17:33.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:17:33.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:17:33.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:17:33.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:17:33.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:17:33.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:17:33.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:17:33.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:17:33.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:17:33.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:17:33.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:17:33.656 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:17:33.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:17:33.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:17:34.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:17:34.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:17:34.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:17:34.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1087: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2024-08-19T15:17:34.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-08-19T15:17:34.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:17:34.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:17:34.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:17:34.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:17:34.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-08-19T15:17:34.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:17:34.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:17:34.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:17:34.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:17:34.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:17:34.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:17:34.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:17:34.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:17:34.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:17:34.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:17:34.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:17:34.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-08-19T15:17:34.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:17:34.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:17:34.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:17:34.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:17:34.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:17:34.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-08-19T15:17:35.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:17:35.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:17:35.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:17:35.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:17:35.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:17:35.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:17:35.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:17:35.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:17:35.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:17:35.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:17:35.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:17:35.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:17:35.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:17:35.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:17:35.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:17:35.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:17:35.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:17:35.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:17:35.989 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:17:35.990 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:17:35.990 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:17:35.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:17:35.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:17:35.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:17:35.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:17:35.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:17:35.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:17:35.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:17:35.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:17:35.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:17:35.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:17:35.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:17:35.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:17:35.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:17:35.992 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:17:35.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:17:35.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:17:35.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:17:35.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:17:35.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:17:35.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:17:36.020 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:36.019+0000 7f6cd240f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:17:36.031 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:36.030+0000 7f6cd240f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:17:36.032 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:36.031+0000 7f6cd240f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:17:36.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:17:36.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:17:36.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:17:36.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:17:36.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:17:36.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:36.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:17:36.277 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:17:36.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:36.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:17:36.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:17:36.556 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:36.554+0000 7f6cd240f640 -1 Falling back to public interface 2024-08-19T15:17:37.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:17:37.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:37.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:17:37.544 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:17:37.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:37.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:17:37.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:17:38.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:17:38.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:38.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:17:38.820 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:17:38.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:38.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:17:38.886 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:38.884+0000 7f6cd240f640 -1 osd.1 112 log_to_monitors true 2024-08-19T15:17:39.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:17:40.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:17:40.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:40.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:17:40.141 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:17:40.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:40.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:17:40.460 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 116 up_thru 116 down_at 113 last_clean_interval [110,112) [v2:127.0.0.1:6810/1630565016,v1:127.0.0.1:6811/1630565016] [v2:127.0.0.1:6812/1630565016,v1:127.0.0.1:6813/1630565016] exists,up 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:17:40.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:17:40.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:17:40.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:17:40.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:17:40.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:17:40.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:17:40.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:17:40.461 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:17:40.461 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:17:40.461 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:17:40.461 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:17:40.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:17:40.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:17:40.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:17:40.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:17:40.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:17:40.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:17:40.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:17:40.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:17:40.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:17:40.853 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:17:40.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:17:40.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:17:40.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:17:40.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566088 2024-08-19T15:17:40.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566088 2024-08-19T15:17:40.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566088' 2024-08-19T15:17:40.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:17:40.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:17:41.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206338 2024-08-19T15:17:41.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206338 2024-08-19T15:17:41.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566088 1-498216206338' 2024-08-19T15:17:41.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:17:41.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566088 2024-08-19T15:17:41.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:17:41.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:17:41.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566088 2024-08-19T15:17:41.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:17:41.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566088 2024-08-19T15:17:41.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566088' 2024-08-19T15:17:41.061 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 450971566088 2024-08-19T15:17:41.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:17:41.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566087 -lt 450971566088 2024-08-19T15:17:41.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:17:42.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:17:42.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:17:42.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566088 -lt 450971566088 2024-08-19T15:17:42.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:17:42.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-498216206338 2024-08-19T15:17:42.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:17:42.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:17:42.606 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-498216206338 2024-08-19T15:17:42.606 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:17:42.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206338 2024-08-19T15:17:42.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 498216206338' 2024-08-19T15:17:42.606 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 498216206338 2024-08-19T15:17:42.606 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:17:42.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206338 -lt 498216206338 2024-08-19T15:17:42.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:17:42.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:17:42.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:17:43.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:17:43.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:17:43.203 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:17:43.203 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:17:43.203 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:17:43.203 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:17:43.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:17:43.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:17:43.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:17:43.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:17:43.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:17:43.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:17:43.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:17:43.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:17:43.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:17:43.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-08-19T15:17:43.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:17:43.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-08-19T15:17:43.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 9 % 2 2024-08-19T15:17:43.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-08-19T15:17:43.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-08-19T15:17:43.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1093: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-08-19T15:17:43.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:17:43.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:17:43.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:17:43.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:17:43.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-08-19T15:17:43.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:17:43.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:17:43.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:17:43.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:17:43.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:17:43.802 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:17:43.802 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:17:43.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:17:43.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:17:43.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:17:44.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:17:44.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-08-19T15:17:44.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:17:44.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:17:44.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:17:44.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:17:44.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:17:44.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2024-08-19T15:17:45.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:17:45.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:17:45.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:17:45.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:17:45.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:17:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:17:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:17:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:17:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:17:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:17:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:17:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:17:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:17:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:17:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:17:45.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:17:45.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:17:45.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:17:45.713 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:17:45.713 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:17:45.713 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:17:45.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:17:45.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:17:45.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:17:45.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:17:45.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:17:45.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:17:45.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:17:45.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:17:45.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:17:45.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:17:45.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:17:45.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:17:45.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:17:45.715 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:17:45.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:17:45.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:17:45.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:17:45.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:17:45.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:17:45.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:17:45.741 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:45.739+0000 7fa2cea14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:17:45.744 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:45.742+0000 7fa2cea14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:17:45.745 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:45.743+0000 7fa2cea14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:17:45.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:17:45.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:17:45.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:17:45.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:17:45.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:17:45.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:45.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:17:45.994 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:17:45.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:45.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:17:46.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:17:46.264 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:46.263+0000 7fa2cea14640 -1 Falling back to public interface 2024-08-19T15:17:47.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:17:47.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:47.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:17:47.260 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:17:47.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:47.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:17:47.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:17:48.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:17:48.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:48.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:17:48.537 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:17:48.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:48.537 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:48.529+0000 7fa2cea14640 -1 osd.1 117 log_to_monitors true 2024-08-19T15:17:48.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:17:48.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:17:49.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:17:49.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:49.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:17:49.940 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:17:49.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:49.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:17:50.398 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 121 up_thru 121 down_at 118 last_clean_interval [116,117) [v2:127.0.0.1:6810/1807953844,v1:127.0.0.1:6811/1807953844] [v2:127.0.0.1:6812/1807953844,v1:127.0.0.1:6813/1807953844] exists,up 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:17:50.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:17:50.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:17:50.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:17:50.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:17:50.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:17:50.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:17:50.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:17:50.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:17:50.399 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:17:50.399 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:17:50.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:17:50.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:17:50.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:17:50.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:17:50.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:17:50.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:17:50.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:17:50.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:17:50.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:17:50.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:17:50.751 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:17:50.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:17:50.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:17:50.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:17:50.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566091 2024-08-19T15:17:50.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566091 2024-08-19T15:17:50.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566091' 2024-08-19T15:17:50.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:17:50.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:17:50.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=519691042818 2024-08-19T15:17:50.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 519691042818 2024-08-19T15:17:50.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566091 1-519691042818' 2024-08-19T15:17:50.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:17:50.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566091 2024-08-19T15:17:50.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:17:51.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:17:51.000 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566091 2024-08-19T15:17:51.000 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:17:51.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566091 2024-08-19T15:17:51.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566091' 2024-08-19T15:17:51.002 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 450971566091 2024-08-19T15:17:51.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:17:51.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566090 -lt 450971566091 2024-08-19T15:17:51.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:17:52.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:17:52.269 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:17:52.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566091 -lt 450971566091 2024-08-19T15:17:52.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:17:52.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-519691042818 2024-08-19T15:17:52.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:17:52.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:17:52.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-519691042818 2024-08-19T15:17:52.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:17:52.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=519691042818 2024-08-19T15:17:52.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 519691042818' 2024-08-19T15:17:52.545 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 519691042818 2024-08-19T15:17:52.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:17:52.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 519691042818 -lt 519691042818 2024-08-19T15:17:52.813 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:17:52.813 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:17:52.813 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:17:53.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:17:53.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:17:53.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:17:53.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:17:53.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:17:53.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:17:53.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:17:53.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:17:53.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:17:53.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:17:53.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:17:53.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:17:53.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:17:53.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:17:53.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:17:53.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: echo -n D 2024-08-19T15:17:53.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1095: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-08-19T15:17:53.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1096: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-08-19T15:17:53.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:17:53.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:17:53.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:17:53.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:17:53.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-08-19T15:17:53.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:17:53.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:17:53.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:17:53.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:17:53.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:17:53.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:17:53.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:17:53.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:17:53.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:17:53.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:17:54.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:17:54.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-08-19T15:17:54.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:17:54.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:17:54.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:17:54.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:17:54.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:17:54.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-08-19T15:17:55.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:17:55.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:17:55.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:17:55.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:17:55.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:17:55.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:17:55.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:17:55.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:17:55.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:17:55.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:17:55.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:17:55.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:17:55.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:17:55.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:17:55.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:17:55.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:17:55.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:17:55.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:17:55.966 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:17:55.966 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:17:55.966 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:17:55.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:17:55.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:17:55.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:17:55.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:17:55.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:17:55.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:17:55.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:17:55.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:17:55.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:17:55.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:17:55.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:17:55.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:17:55.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:17:55.969 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:17:55.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:17:55.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:17:55.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:17:55.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:17:55.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:17:55.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:17:55.996 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:55.995+0000 7f065a60e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:17:56.041 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:56.040+0000 7f065a60e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:17:56.042 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:56.041+0000 7f065a60e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:17:56.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:17:56.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:17:56.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:17:56.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:17:56.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:17:56.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:56.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:17:56.246 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:17:56.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:56.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:17:56.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:17:56.564 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:56.563+0000 7f065a60e640 -1 Falling back to public interface 2024-08-19T15:17:57.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:17:57.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:57.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:17:57.534 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:17:57.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:57.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:17:57.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:17:58.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:17:58.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:17:58.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:17:58.781 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:17:58.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:17:58.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:17:58.819 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:17:58.817+0000 7f065a60e640 -1 osd.1 123 log_to_monitors true 2024-08-19T15:17:59.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:18:00.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:18:00.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:00.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:18:00.059 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:18:00.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:00.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:18:00.399 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 127 up_thru 127 down_at 124 last_clean_interval [121,123) [v2:127.0.0.1:6810/3190280880,v1:127.0.0.1:6811/3190280880] [v2:127.0.0.1:6812/3190280880,v1:127.0.0.1:6813/3190280880] exists,up 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:18:00.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:18:00.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:18:00.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:18:00.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:18:00.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:18:00.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:18:00.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:18:00.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:18:00.400 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:18:00.400 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:18:00.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:18:00.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:18:00.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:18:00.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:18:00.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:18:00.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:18:00.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:18:00.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:18:00.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:18:00.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:18:00.759 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:18:00.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:18:00.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:18:00.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:18:00.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566094 2024-08-19T15:18:00.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566094 2024-08-19T15:18:00.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566094' 2024-08-19T15:18:00.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:18:00.862 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:18:00.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=545460846594 2024-08-19T15:18:00.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 545460846594 2024-08-19T15:18:00.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566094 1-545460846594' 2024-08-19T15:18:00.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:18:00.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566094 2024-08-19T15:18:00.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:18:00.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:18:00.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566094 2024-08-19T15:18:00.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:18:00.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566094 2024-08-19T15:18:00.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566094' 2024-08-19T15:18:00.968 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 450971566094 2024-08-19T15:18:00.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:18:01.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566093 -lt 450971566094 2024-08-19T15:18:01.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:18:02.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:18:02.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:18:02.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566094 -lt 450971566094 2024-08-19T15:18:02.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:18:02.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-545460846594 2024-08-19T15:18:02.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:18:02.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:18:02.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-545460846594 2024-08-19T15:18:02.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:18:02.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=545460846594 2024-08-19T15:18:02.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 545460846594' 2024-08-19T15:18:02.515 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 545460846594 2024-08-19T15:18:02.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:18:02.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 545460846594 -lt 545460846594 2024-08-19T15:18:02.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:18:02.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:18:02.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:18:03.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:18:03.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:18:03.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:18:03.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:18:03.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:18:03.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:18:03.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:18:03.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:18:03.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:18:03.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:18:03.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:18:03.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:18:03.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:18:03.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:18:03.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:18:03.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-08-19T15:18:03.704 INFO:tasks.workunit.client.0.smithi102.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-08-19T15:18:03.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:18:03.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-08-19T15:18:03.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 10 % 2 2024-08-19T15:18:03.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-08-19T15:18:03.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-08-19T15:18:03.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:18:03.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-08-19T15:18:03.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 11 % 2 2024-08-19T15:18:03.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-08-19T15:18:03.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-08-19T15:18:03.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:18:03.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-08-19T15:18:03.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 12 % 2 2024-08-19T15:18:03.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-08-19T15:18:03.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-08-19T15:18:03.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:18:03.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-08-19T15:18:03.709 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 13 % 2 2024-08-19T15:18:03.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-08-19T15:18:03.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-08-19T15:18:03.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:18:03.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-08-19T15:18:03.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 14 % 2 2024-08-19T15:18:03.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-08-19T15:18:03.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-08-19T15:18:03.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-08-19T15:18:03.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1107: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-08-19T15:18:03.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:18:03.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:18:03.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:18:03.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:18:03.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-08-19T15:18:03.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:18:03.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:18:03.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:18:03.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:18:03.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:18:03.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:18:03.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:18:03.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:18:03.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:18:03.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:18:04.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:18:04.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-08-19T15:18:04.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:18:04.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:18:04.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:18:04.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:18:04.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:18:04.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-08-19T15:18:05.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:18:05.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:18:05.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:18:05.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:18:05.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:18:05.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:18:05.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:18:05.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:18:05.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:18:05.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:18:05.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:18:05.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:18:05.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:18:05.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:18:05.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:18:05.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:18:05.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:18:05.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:18:05.570 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:18:05.570 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:18:05.570 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:18:05.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:18:05.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:18:05.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:18:05.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:18:05.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:18:05.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:18:05.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:18:05.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:18:05.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:18:05.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:18:05.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:18:05.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:18:05.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:18:05.572 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:18:05.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:18:05.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:18:05.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:18:05.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:18:05.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:18:05.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:18:05.599 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:05.598+0000 7fc181816640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:18:05.601 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:05.600+0000 7fc181816640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:18:05.603 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:05.602+0000 7fc181816640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:18:05.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:18:05.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:18:05.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:18:05.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:18:05.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:18:05.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:05.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:18:05.850 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:18:05.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:05.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:18:06.120 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:06.119+0000 7fc181816640 -1 Falling back to public interface 2024-08-19T15:18:06.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:18:07.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:18:07.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:07.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:18:07.141 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:18:07.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:07.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:18:07.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:18:08.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:18:08.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:08.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:18:08.416 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:18:08.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:08.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:18:08.463 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:08.462+0000 7fc181816640 -1 osd.0 129 log_to_monitors true 2024-08-19T15:18:08.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:18:09.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:18:09.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:09.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:18:09.726 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:18:09.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:09.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:18:10.003 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 133 up_thru 133 down_at 130 last_clean_interval [105,129) [v2:127.0.0.1:6802/1033019101,v1:127.0.0.1:6803/1033019101] [v2:127.0.0.1:6804/1033019101,v1:127.0.0.1:6805/1033019101] exists,up a9a4af16-d461-4c2f-ab0d-b99df37a8a3e 2024-08-19T15:18:10.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:18:10.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:18:10.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:18:10.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:18:10.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:18:10.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:18:10.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:18:10.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:18:10.005 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:18:10.005 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:18:10.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:18:10.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:18:10.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:18:10.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:18:10.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:18:10.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:18:10.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:18:10.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:18:10.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:18:10.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:18:10.359 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:18:10.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:18:10.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:18:10.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:18:10.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=571230650370 2024-08-19T15:18:10.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 571230650370 2024-08-19T15:18:10.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650370' 2024-08-19T15:18:10.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:18:10.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:18:10.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=545460846597 2024-08-19T15:18:10.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 545460846597 2024-08-19T15:18:10.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650370 1-545460846597' 2024-08-19T15:18:10.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:18:10.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-571230650370 2024-08-19T15:18:10.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:18:10.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:18:10.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-571230650370 2024-08-19T15:18:10.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:18:10.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=571230650370 2024-08-19T15:18:10.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 571230650370' 2024-08-19T15:18:10.570 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 571230650370 2024-08-19T15:18:10.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:18:10.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 571230650370 2024-08-19T15:18:10.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:18:11.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:18:11.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:18:12.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650370 -lt 571230650370 2024-08-19T15:18:12.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:18:12.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-545460846597 2024-08-19T15:18:12.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:18:12.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:18:12.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-545460846597 2024-08-19T15:18:12.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:18:12.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=545460846597 2024-08-19T15:18:12.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 545460846597' 2024-08-19T15:18:12.173 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 545460846597 2024-08-19T15:18:12.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:18:12.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 545460846597 -lt 545460846597 2024-08-19T15:18:12.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:18:12.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:18:12.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:18:12.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:18:12.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:18:12.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:18:12.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:18:12.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:18:12.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:18:12.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:18:12.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:18:13.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:18:13.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:18:13.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:18:13.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:18:13.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:18:13.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:18:13.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:18:13.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1108: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-08-19T15:18:13.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:18:13.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:18:13.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:18:13.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:18:13.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-08-19T15:18:13.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:18:13.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:18:13.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:18:13.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:18:13.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:18:13.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:18:13.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:18:13.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:18:13.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:18:13.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:18:13.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:18:13.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-08-19T15:18:13.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:18:13.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:18:13.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:18:13.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:18:13.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:18:13.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2024-08-19T15:18:15.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:18:15.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:18:15.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:18:15.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:18:15.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:18:15.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:18:15.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:18:15.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:18:15.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:18:15.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:18:15.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:18:15.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:18:15.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:18:15.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:18:15.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:18:15.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:18:15.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:18:15.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:18:15.295 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:18:15.295 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:18:15.296 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:18:15.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:18:15.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:18:15.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:18:15.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:18:15.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:18:15.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:18:15.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:18:15.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:18:15.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:18:15.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:18:15.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:18:15.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:18:15.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:18:15.297 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:18:15.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:18:15.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:18:15.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:18:15.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:18:15.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:18:15.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:18:15.324 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:15.323+0000 7fb6f2f1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:18:15.326 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:15.325+0000 7fb6f2f1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:18:15.328 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:15.327+0000 7fb6f2f1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:18:15.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:18:15.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:18:15.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:18:15.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:18:15.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:18:15.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:15.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:18:15.579 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:18:15.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:15.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:18:15.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:18:15.853 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:15.852+0000 7fb6f2f1f640 -1 Falling back to public interface 2024-08-19T15:18:16.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:18:16.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:16.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:18:16.852 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:18:16.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:16.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:18:17.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:18:18.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:18:18.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:18.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:18:18.130 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:18:18.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:18.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:18:18.315 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:18.314+0000 7fb6f2f1f640 -1 osd.1 134 log_to_monitors true 2024-08-19T15:18:18.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:18:19.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:18:19.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:19.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:18:19.435 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:18:19.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:19.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:18:20.030 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 138 up_thru 138 down_at 135 last_clean_interval [127,134) [v2:127.0.0.1:6810/2362261728,v1:127.0.0.1:6811/2362261728] [v2:127.0.0.1:6812/2362261728,v1:127.0.0.1:6813/2362261728] exists,up 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:18:20.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:18:20.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:18:20.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:18:20.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:18:20.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:18:20.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:18:20.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:18:20.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:18:20.030 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:18:20.030 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:18:20.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:18:20.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:18:20.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:18:20.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:18:20.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:18:20.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:18:20.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:18:20.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:18:20.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:18:20.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:18:20.391 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:18:20.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:18:20.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:18:20.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:18:20.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=571230650373 2024-08-19T15:18:20.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 571230650373 2024-08-19T15:18:20.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650373' 2024-08-19T15:18:20.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:18:20.494 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:18:20.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=592705486850 2024-08-19T15:18:20.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 592705486850 2024-08-19T15:18:20.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650373 1-592705486850' 2024-08-19T15:18:20.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:18:20.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-571230650373 2024-08-19T15:18:20.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:18:20.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:18:20.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-571230650373 2024-08-19T15:18:20.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:18:20.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=571230650373 2024-08-19T15:18:20.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 571230650373' 2024-08-19T15:18:20.602 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 571230650373 2024-08-19T15:18:20.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:18:20.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650372 -lt 571230650373 2024-08-19T15:18:20.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:18:21.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:18:21.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:18:22.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650373 -lt 571230650373 2024-08-19T15:18:22.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:18:22.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-592705486850 2024-08-19T15:18:22.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:18:22.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:18:22.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-592705486850 2024-08-19T15:18:22.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:18:22.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=592705486850 2024-08-19T15:18:22.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 592705486850' 2024-08-19T15:18:22.171 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 592705486850 2024-08-19T15:18:22.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:18:22.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 592705486850 -lt 592705486850 2024-08-19T15:18:22.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:18:22.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:18:22.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:18:22.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:18:22.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:18:22.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:18:22.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:18:22.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:18:22.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:18:22.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:18:22.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:18:23.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:18:23.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:18:23.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:18:23.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:18:23.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:18:23.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:18:23.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:18:23.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1109: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2024-08-19T15:18:23.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:18:23.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-08-19T15:18:23.363 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 15 % 2 2024-08-19T15:18:23.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-08-19T15:18:23.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-08-19T15:18:23.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1113: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-08-19T15:18:23.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:18:23.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:18:23.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:18:23.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:18:23.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-08-19T15:18:23.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:18:23.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:18:23.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:18:23.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:18:23.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:18:23.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:18:23.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:18:23.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:18:23.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:18:23.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:18:23.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:18:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-08-19T15:18:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:18:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:18:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:18:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:18:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:18:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2024-08-19T15:18:24.678 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-08-19T15:18:25.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:18:25.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:18:25.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:18:25.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:18:25.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:18:25.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:18:25.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:18:25.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:18:25.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:18:25.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:18:25.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:18:25.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:18:25.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:18:25.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:18:25.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:18:25.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:18:25.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:18:25.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:18:25.204 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:18:25.204 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:18:25.204 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:18:25.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:18:25.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:18:25.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:18:25.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:18:25.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:18:25.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:18:25.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:18:25.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:18:25.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:18:25.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:18:25.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:18:25.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:18:25.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:18:25.206 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:18:25.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:18:25.207 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:18:25.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:18:25.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:18:25.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:18:25.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:18:25.233 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:25.232+0000 7f1c32215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:18:25.235 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:25.234+0000 7f1c32215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:18:25.236 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:25.235+0000 7f1c32215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:18:25.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:18:25.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:18:25.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:18:25.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:18:25.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:18:25.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:25.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:18:25.480 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:18:25.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:25.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:18:25.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:18:25.761 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:25.761+0000 7f1c32215640 -1 Falling back to public interface 2024-08-19T15:18:26.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:18:26.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:26.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:18:26.753 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:18:26.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:26.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:18:27.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:18:28.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:18:28.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:28.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:18:28.026 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:18:28.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:28.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:18:28.156 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:28.155+0000 7f1c32215640 -1 osd.1 140 log_to_monitors true 2024-08-19T15:18:28.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:18:29.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:18:29.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:29.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:18:29.332 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:18:29.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:29.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:18:29.602 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 144 up_thru 144 down_at 141 last_clean_interval [138,140) [v2:127.0.0.1:6810/2682836668,v1:127.0.0.1:6811/2682836668] [v2:127.0.0.1:6812/2682836668,v1:127.0.0.1:6813/2682836668] exists,up 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:18:29.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:18:29.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:18:29.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:18:29.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:18:29.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:18:29.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:18:29.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:18:29.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:18:29.603 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:18:29.603 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:18:29.604 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:18:29.604 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:18:29.604 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:18:29.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:18:29.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:18:29.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:18:29.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:18:29.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:18:29.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:18:29.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:18:29.961 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:18:29.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:18:29.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:18:29.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:18:30.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=571230650376 2024-08-19T15:18:30.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 571230650376 2024-08-19T15:18:30.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650376' 2024-08-19T15:18:30.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:18:30.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:18:30.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=618475290626 2024-08-19T15:18:30.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 618475290626 2024-08-19T15:18:30.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650376 1-618475290626' 2024-08-19T15:18:30.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:18:30.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-571230650376 2024-08-19T15:18:30.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:18:30.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:18:30.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-571230650376 2024-08-19T15:18:30.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:18:30.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=571230650376 2024-08-19T15:18:30.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 571230650376' 2024-08-19T15:18:30.170 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 571230650376 2024-08-19T15:18:30.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:18:30.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650375 -lt 571230650376 2024-08-19T15:18:30.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:18:31.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:18:31.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:18:31.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650375 -lt 571230650376 2024-08-19T15:18:31.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:18:32.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:18:32.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:18:32.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650376 -lt 571230650376 2024-08-19T15:18:32.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:18:32.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-618475290626 2024-08-19T15:18:32.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:18:32.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:18:32.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-618475290626 2024-08-19T15:18:32.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:18:32.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=618475290626 2024-08-19T15:18:32.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 618475290626' 2024-08-19T15:18:32.985 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 618475290626 2024-08-19T15:18:32.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:18:33.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 618475290626 -lt 618475290626 2024-08-19T15:18:33.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:18:33.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:18:33.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:18:33.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:18:33.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:18:33.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:18:33.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:18:33.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:18:33.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:18:33.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:18:33.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:18:33.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:18:33.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:18:33.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:18:33.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:18:34.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:18:34.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:18:34.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:18:34.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:18:34.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-08-19T15:18:34.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 16 % 2 2024-08-19T15:18:34.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-08-19T15:18:34.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-08-19T15:18:34.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1117: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-08-19T15:18:34.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:18:34.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:18:34.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:18:34.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:18:34.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-08-19T15:18:34.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:18:34.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:18:34.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:18:34.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:18:34.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:18:34.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:18:34.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:18:34.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:18:34.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:18:34.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:18:34.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:18:34.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-08-19T15:18:34.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:18:34.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:18:34.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:18:34.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:18:34.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:18:34.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2024-08-19T15:18:35.463 INFO:tasks.workunit.client.0.smithi102.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-08-19T15:18:35.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:18:35.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:18:35.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:18:35.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:18:35.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:18:35.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:18:35.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:18:35.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:18:35.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:18:35.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:18:35.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:18:35.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:18:35.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:18:35.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:18:35.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:18:35.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:18:35.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:18:35.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:18:35.984 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:18:35.984 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:18:35.984 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:18:35.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:18:35.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:18:35.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:18:35.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:18:35.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:18:35.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:18:35.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:18:35.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:18:35.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:18:35.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:18:35.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:18:35.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:18:35.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:18:35.986 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:18:35.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:18:35.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:18:35.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:18:35.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:18:35.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:18:35.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:18:36.014 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:36.012+0000 7f497a884640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:18:36.025 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:36.023+0000 7f497a884640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:18:36.027 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:36.025+0000 7f497a884640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:18:36.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:18:36.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:18:36.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:18:36.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:18:36.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:18:36.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:36.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:18:36.263 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:18:36.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:36.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:18:36.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:18:36.555 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:36.553+0000 7f497a884640 -1 Falling back to public interface 2024-08-19T15:18:37.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:18:37.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:37.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:18:37.529 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:18:37.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:37.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:18:37.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:18:38.755 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:38.754+0000 7f497a884640 -1 osd.0 145 log_to_monitors true 2024-08-19T15:18:38.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:18:38.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:38.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:18:38.803 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:18:38.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:38.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:18:39.152 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 149 up_thru 144 down_at 146 last_clean_interval [133,145) [v2:127.0.0.1:6802/2777259017,v1:127.0.0.1:6803/2777259017] [v2:127.0.0.1:6804/2777259017,v1:127.0.0.1:6805/2777259017] exists,up a9a4af16-d461-4c2f-ab0d-b99df37a8a3e 2024-08-19T15:18:39.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:18:39.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:18:39.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:18:39.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:18:39.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:18:39.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:18:39.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:18:39.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:18:39.152 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:18:39.152 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:18:39.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:18:39.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:18:39.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:18:39.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:18:39.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:18:39.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:18:39.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:18:39.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:18:39.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:18:39.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:18:39.662 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:18:39.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:18:39.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:18:39.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:18:39.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=639950127106 2024-08-19T15:18:39.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 639950127106 2024-08-19T15:18:39.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127106' 2024-08-19T15:18:39.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:18:39.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:18:39.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=618475290629 2024-08-19T15:18:39.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 618475290629 2024-08-19T15:18:39.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127106 1-618475290629' 2024-08-19T15:18:39.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:18:39.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-639950127106 2024-08-19T15:18:39.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:18:39.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:18:39.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-639950127106 2024-08-19T15:18:39.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:18:39.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=639950127106 2024-08-19T15:18:39.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 639950127106' 2024-08-19T15:18:39.882 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 639950127106 2024-08-19T15:18:39.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:18:40.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127106 -lt 639950127106 2024-08-19T15:18:40.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:18:40.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-618475290629 2024-08-19T15:18:40.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:18:40.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:18:40.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-618475290629 2024-08-19T15:18:40.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:18:40.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=618475290629 2024-08-19T15:18:40.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 618475290629' 2024-08-19T15:18:40.159 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 618475290629 2024-08-19T15:18:40.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:18:40.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 618475290628 -lt 618475290629 2024-08-19T15:18:40.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:18:41.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:18:41.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:18:41.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 618475290628 -lt 618475290629 2024-08-19T15:18:41.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:18:42.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:18:42.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:18:42.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 618475290629 -lt 618475290629 2024-08-19T15:18:42.979 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:18:42.979 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:18:42.979 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:18:43.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:18:43.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:18:43.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:18:43.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:18:43.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:18:43.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:18:43.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:18:43.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:18:43.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-08-19T15:18:43.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:18:43.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:18:43.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:18:43.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-08-19T15:18:43.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-08-19T15:18:43.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T15:18:43.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-08-19T15:18:43.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T15:18:43.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T15:18:44.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T15:18:44.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:18:44.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:18:44.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:18:44.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:18:44.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:18:44.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:18:44.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:18:44.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-08-19T15:18:44.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:18:44.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:18:44.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:18:44.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-08-19T15:18:44.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-08-19T15:18:44.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T15:18:44.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T15:18:44.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T15:18:44.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T15:18:44.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T15:18:44.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T15:18:44.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T15:18:45.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T15:18:45.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T15:18:45.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-19T15:18:45.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T15:18:45.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-19T15:18:45.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T15:18:45.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:18:45.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:18:45.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:18:45.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:18:45.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:18:45.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:18:45.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:18:45.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-08-19T15:18:45.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:18:45.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:18:45.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:18:45.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-08-19T15:18:45.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-08-19T15:18:45.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T15:18:45.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T15:18:45.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T15:18:45.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T15:18:45.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T15:18:45.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T15:18:45.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T15:18:46.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T15:18:46.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T15:18:46.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-08-19T15:18:46.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T15:18:46.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-19T15:18:46.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T15:18:46.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:18:46.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:18:46.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:18:46.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:18:46.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:18:46.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:18:46.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:18:46.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:18:46.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:18:46.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:18:46.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:18:47.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:18:47.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:18:47.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:18:47.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1118: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-08-19T15:18:47.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1119: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-08-19T15:18:47.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:18:47.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:18:47.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:18:47.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:18:47.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-08-19T15:18:47.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:18:47.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:18:47.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:18:47.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:18:47.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:18:47.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:18:47.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:18:47.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:18:47.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:18:47.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:18:47.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:18:47.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-08-19T15:18:47.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:18:47.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:18:47.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:18:47.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:18:47.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:18:47.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-08-19T15:18:48.658 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-08-19T15:18:48.658 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-08-19T15:18:49.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:18:49.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:18:49.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:18:49.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:18:49.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:18:49.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:18:49.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:18:49.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:18:49.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:18:49.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:18:49.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:18:49.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:18:49.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:18:49.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:18:49.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:18:49.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:18:49.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:18:49.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:18:49.178 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:18:49.179 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:18:49.179 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:18:49.179 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:18:49.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:18:49.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:18:49.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:18:49.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:18:49.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:18:49.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:18:49.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:18:49.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:18:49.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:18:49.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:18:49.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:18:49.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:18:49.180 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:18:49.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:18:49.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:18:49.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:18:49.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:18:49.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:18:49.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:18:49.205 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:49.204+0000 7f93fed57640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:18:49.207 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:49.206+0000 7f93fed57640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:18:49.208 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:49.207+0000 7f93fed57640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:18:49.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:18:49.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:18:49.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:18:49.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:18:49.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:18:49.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:49.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:18:49.449 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:18:49.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:49.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:18:49.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:18:49.735 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:49.734+0000 7f93fed57640 -1 Falling back to public interface 2024-08-19T15:18:50.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:18:50.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:50.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:18:50.715 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:18:50.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:50.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:18:50.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:18:51.942 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:18:51.941+0000 7f93fed57640 -1 osd.1 150 log_to_monitors true 2024-08-19T15:18:51.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:18:51.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:51.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:18:51.987 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:18:51.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:51.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:18:52.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:18:53.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:18:53.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:18:53.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:18:53.262 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:18:53.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:18:53.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:18:53.548 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 154 up_thru 154 down_at 151 last_clean_interval [144,150) [v2:127.0.0.1:6810/1690087285,v1:127.0.0.1:6811/1690087285] [v2:127.0.0.1:6812/1690087285,v1:127.0.0.1:6813/1690087285] exists,up 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:18:53.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:18:53.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:18:53.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:18:53.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:18:53.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:18:53.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:18:53.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:18:53.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:18:53.549 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:18:53.549 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:18:53.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:18:53.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:18:53.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:18:53.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:18:53.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:18:53.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:18:53.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:18:53.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:18:53.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:18:53.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:18:53.911 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:18:53.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:18:53.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:18:53.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:18:54.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=639950127110 2024-08-19T15:18:54.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 639950127110 2024-08-19T15:18:54.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127110' 2024-08-19T15:18:54.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:18:54.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:18:54.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=661424963586 2024-08-19T15:18:54.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 661424963586 2024-08-19T15:18:54.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127110 1-661424963586' 2024-08-19T15:18:54.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:18:54.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-639950127110 2024-08-19T15:18:54.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:18:54.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:18:54.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-639950127110 2024-08-19T15:18:54.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:18:54.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=639950127110 2024-08-19T15:18:54.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 639950127110' 2024-08-19T15:18:54.120 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 639950127110 2024-08-19T15:18:54.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:18:54.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127108 -lt 639950127110 2024-08-19T15:18:54.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:18:55.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:18:55.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:18:55.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127108 -lt 639950127110 2024-08-19T15:18:55.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:18:56.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:18:56.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:18:56.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127110 -lt 639950127110 2024-08-19T15:18:56.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:18:56.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-661424963586 2024-08-19T15:18:56.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:18:56.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:18:56.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-661424963586 2024-08-19T15:18:56.939 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:18:56.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=661424963586 2024-08-19T15:18:56.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 661424963586' 2024-08-19T15:18:56.943 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 661424963586 2024-08-19T15:18:56.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:18:57.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 661424963586 -lt 661424963586 2024-08-19T15:18:57.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:18:57.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:18:57.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:18:57.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:18:57.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:18:57.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:18:57.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:18:57.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:18:57.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:18:57.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:18:57.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:18:57.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:18:57.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:18:57.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:18:57.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:18:58.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:18:58.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:18:58.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:18:58.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:18:58.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-08-19T15:18:58.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 17 % 2 2024-08-19T15:18:58.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-08-19T15:18:58.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-08-19T15:18:58.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1124: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2024-08-19T15:18:58.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1125: TEST_corrupt_scrub_replicated: echo ROBJ17 2024-08-19T15:18:58.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-08-19T15:18:58.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:18:58.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:18:58.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:18:58.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:18:58.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-08-19T15:18:58.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:18:58.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:18:58.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:18:58.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:18:58.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:18:58.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:18:58.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:18:58.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:18:58.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:18:58.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:18:58.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:18:58.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-08-19T15:18:58.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:18:58.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:18:58.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:18:58.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:18:58.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:18:58.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-08-19T15:18:59.497 INFO:tasks.workunit.client.0.smithi102.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-08-19T15:19:00.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:19:00.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:19:00.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:19:00.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:19:00.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:19:00.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:19:00.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:19:00.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:19:00.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:19:00.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:19:00.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:19:00.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:19:00.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:19:00.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:19:00.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:19:00.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:19:00.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:19:00.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:19:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:19:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:19:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:19:00.091 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:19:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:19:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:19:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:19:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:19:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:19:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:19:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:19:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:19:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:19:00.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:19:00.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:19:00.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:19:00.092 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:19:00.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:19:00.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:19:00.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:19:00.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:19:00.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:19:00.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:19:00.120 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:00.119+0000 7f04b0430640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:19:00.122 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:00.121+0000 7f04b0430640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:19:00.124 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:00.123+0000 7f04b0430640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:19:00.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:19:00.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:19:00.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:19:00.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:19:00.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:19:00.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:19:00.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:19:00.372 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:19:00.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:19:00.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:19:00.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:19:00.647 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:00.647+0000 7f04b0430640 -1 Falling back to public interface 2024-08-19T15:19:01.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:19:01.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:19:01.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:19:01.638 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:19:01.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:19:01.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:19:01.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:19:02.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:02.860+0000 7f04b0430640 -1 osd.0 156 log_to_monitors true 2024-08-19T15:19:02.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:19:02.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:19:02.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:19:02.920 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:19:02.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:19:02.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:19:03.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:19:04.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:19:04.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:19:04.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:19:04.300 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:19:04.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:19:04.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:19:04.583 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 160 up_thru 160 down_at 157 last_clean_interval [149,156) [v2:127.0.0.1:6802/1601786727,v1:127.0.0.1:6803/1601786727] [v2:127.0.0.1:6804/1601786727,v1:127.0.0.1:6805/1601786727] exists,up a9a4af16-d461-4c2f-ab0d-b99df37a8a3e 2024-08-19T15:19:04.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:19:04.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:19:04.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:19:04.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:19:04.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:19:04.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:19:04.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:19:04.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:19:04.585 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:19:04.585 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:19:04.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:19:04.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:19:04.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:19:04.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:19:04.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:19:04.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:19:04.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:19:04.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:19:04.678 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:19:04.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:19:04.942 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:19:04.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:19:04.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:19:04.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:19:05.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=687194767362 2024-08-19T15:19:05.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 687194767362 2024-08-19T15:19:05.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767362' 2024-08-19T15:19:05.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:19:05.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:19:05.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=661424963589 2024-08-19T15:19:05.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 661424963589 2024-08-19T15:19:05.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767362 1-661424963589' 2024-08-19T15:19:05.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:19:05.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-687194767362 2024-08-19T15:19:05.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:19:05.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:19:05.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-687194767362 2024-08-19T15:19:05.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:19:05.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=687194767362 2024-08-19T15:19:05.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 687194767362' 2024-08-19T15:19:05.192 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 687194767362 2024-08-19T15:19:05.192 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:19:05.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 687194767362 2024-08-19T15:19:05.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:19:06.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:19:06.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:19:06.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767362 -lt 687194767362 2024-08-19T15:19:06.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:19:06.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-661424963589 2024-08-19T15:19:06.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:19:06.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:19:06.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-661424963589 2024-08-19T15:19:06.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:19:06.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=661424963589 2024-08-19T15:19:06.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 661424963589' 2024-08-19T15:19:06.734 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 661424963589 2024-08-19T15:19:06.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:19:07.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 661424963589 -lt 661424963589 2024-08-19T15:19:07.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:19:07.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:19:07.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:19:07.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:19:07.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:19:07.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:19:07.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:19:07.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:19:07.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:19:07.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:19:07.333 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:19:07.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:19:07.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:19:07.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:19:07.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:19:07.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:19:07.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:19:07.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:19:07.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1127: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-08-19T15:19:07.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:19:07.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:19:07.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:19:07.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:19:07.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-08-19T15:19:07.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:19:07.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:19:07.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:19:07.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:19:07.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:19:07.929 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:19:07.929 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:19:07.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:19:07.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:19:07.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:19:08.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:19:08.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-08-19T15:19:08.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:19:08.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:19:08.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:19:08.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:19:08.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:19:08.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-08-19T15:19:08.955 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-08-19T15:19:08.955 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-08-19T15:19:09.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:19:09.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:19:09.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:19:09.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:19:09.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:19:09.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:19:09.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:19:09.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:19:09.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:19:09.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:19:09.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:19:09.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:19:09.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:19:09.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:19:09.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:19:09.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:19:09.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:19:09.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:19:09.504 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:19:09.504 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:19:09.504 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:19:09.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:19:09.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:19:09.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:19:09.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:19:09.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:19:09.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:19:09.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:19:09.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:19:09.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:19:09.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:19:09.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:19:09.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:19:09.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:19:09.505 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:19:09.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:19:09.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:19:09.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:19:09.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:19:09.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:19:09.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:19:09.531 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:09.530+0000 7f27d381d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:19:09.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:09.532+0000 7f27d381d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:19:09.535 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:09.534+0000 7f27d381d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:19:09.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:19:09.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:19:09.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:19:09.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:19:09.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:19:09.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:19:09.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:19:09.780 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:19:09.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:19:09.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:19:10.051 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:10.050+0000 7f27d381d640 -1 Falling back to public interface 2024-08-19T15:19:10.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:19:11.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:19:11.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:19:11.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:19:11.118 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:19:11.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:19:11.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:19:11.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:19:12.218 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:12.217+0000 7f27d381d640 -1 osd.1 161 log_to_monitors true 2024-08-19T15:19:12.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:19:12.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:19:12.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:19:12.392 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:19:12.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:19:12.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:19:12.738 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 165 up_thru 160 down_at 162 last_clean_interval [154,161) [v2:127.0.0.1:6810/1856863770,v1:127.0.0.1:6811/1856863770] [v2:127.0.0.1:6812/1856863770,v1:127.0.0.1:6813/1856863770] exists,up 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:19:12.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:19:12.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:19:12.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:19:12.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:19:12.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:19:12.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:19:12.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:19:12.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:19:12.739 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:19:12.739 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:19:12.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:19:12.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:19:12.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:19:12.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:19:12.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:19:12.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:19:12.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:19:12.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:19:12.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:19:13.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:19:13.197 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:19:13.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:19:13.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:19:13.197 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:19:13.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=687194767365 2024-08-19T15:19:13.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 687194767365 2024-08-19T15:19:13.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767365' 2024-08-19T15:19:13.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:19:13.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:19:13.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=708669603842 2024-08-19T15:19:13.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 708669603842 2024-08-19T15:19:13.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767365 1-708669603842' 2024-08-19T15:19:13.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:19:13.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-687194767365 2024-08-19T15:19:13.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:19:13.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:19:13.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-687194767365 2024-08-19T15:19:13.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:19:13.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=687194767365 2024-08-19T15:19:13.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 687194767365' 2024-08-19T15:19:13.406 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 687194767365 2024-08-19T15:19:13.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:19:13.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767363 -lt 687194767365 2024-08-19T15:19:13.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:19:14.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:19:14.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:19:14.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767365 -lt 687194767365 2024-08-19T15:19:14.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:19:14.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-708669603842 2024-08-19T15:19:14.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:19:14.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:19:14.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-708669603842 2024-08-19T15:19:14.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:19:14.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=708669603842 2024-08-19T15:19:14.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 708669603842' 2024-08-19T15:19:14.947 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 708669603842 2024-08-19T15:19:14.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:19:15.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 708669603842 -lt 708669603842 2024-08-19T15:19:15.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:19:15.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:19:15.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:19:15.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:19:15.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:19:15.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:19:15.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:19:15.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:19:15.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:19:15.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:19:15.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:19:15.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:19:15.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:19:15.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:19:15.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:19:16.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-08-19T15:19:16.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-08-19T15:19:16.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T15:19:16.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-08-19T15:19:16.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T15:19:16.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T15:19:16.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T15:19:16.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:19:16.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:19:16.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:19:16.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:19:16.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:19:16.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:19:16.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:19:16.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:19:16.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:19:16.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:19:16.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:19:16.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-08-19T15:19:16.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-08-19T15:19:16.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T15:19:16.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T15:19:16.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T15:19:16.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T15:19:16.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T15:19:16.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T15:19:16.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T15:19:17.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T15:19:17.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T15:19:17.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-19T15:19:17.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T15:19:17.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-19T15:19:17.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T15:19:17.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:19:17.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:19:17.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:19:17.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:19:17.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:19:17.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:19:17.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:19:17.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:19:17.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:19:17.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:19:17.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:19:17.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-08-19T15:19:17.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-08-19T15:19:17.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T15:19:17.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T15:19:17.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T15:19:17.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T15:19:17.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T15:19:17.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T15:19:17.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T15:19:18.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T15:19:18.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T15:19:18.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-08-19T15:19:18.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T15:19:18.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-19T15:19:18.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T15:19:18.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:19:18.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:19:18.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:19:18.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:19:18.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:19:18.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:19:18.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:19:18.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:19:18.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:19:18.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:19:18.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:19:19.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-08-19T15:19:19.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-08-19T15:19:19.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T15:19:19.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T15:19:19.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T15:19:19.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T15:19:19.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T15:19:19.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T15:19:19.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T15:19:19.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T15:19:19.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T15:19:19.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-08-19T15:19:19.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T15:19:19.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-08-19T15:19:20.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T15:19:20.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:19:20.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:19:20.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:19:20.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:19:20.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:19:20.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:19:20.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:19:20.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:19:20.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:19:20.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:19:20.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:19:21.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:19:21.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:19:21.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:19:21.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:19:21.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-08-19T15:19:21.046 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 18 % 2 2024-08-19T15:19:21.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-08-19T15:19:21.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-08-19T15:19:21.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1132: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2024-08-19T15:19:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1133: TEST_corrupt_scrub_replicated: echo ROBJ18 2024-08-19T15:19:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-08-19T15:19:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:19:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:19:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:19:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:19:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-08-19T15:19:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:19:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:19:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:19:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:19:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:19:21.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:19:21.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:19:21.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:19:21.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:19:21.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:19:21.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:19:21.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-08-19T15:19:21.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:19:21.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:19:21.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:19:21.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:19:21.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:19:21.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-08-19T15:19:22.607 INFO:tasks.workunit.client.0.smithi102.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-08-19T15:19:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:19:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:19:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:19:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:19:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:19:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:19:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:19:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:19:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:19:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:19:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:19:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:19:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:19:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:19:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:19:23.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:19:23.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:19:23.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:19:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:19:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:19:23.213 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:19:23.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:19:23.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:19:23.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:19:23.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:19:23.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:19:23.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:19:23.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:19:23.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:19:23.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:19:23.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:19:23.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:19:23.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:19:23.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:19:23.216 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:19:23.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:19:23.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:19:23.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:19:23.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:19:23.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:19:23.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:19:23.243 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:23.241+0000 7f828f414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:19:23.245 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:23.243+0000 7f828f414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:19:23.246 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:23.244+0000 7f828f414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:19:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:19:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:19:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:19:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:19:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:19:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:19:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:19:23.489 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:19:23.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:19:23.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:19:23.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:19:23.764 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:23.763+0000 7f828f414640 -1 Falling back to public interface 2024-08-19T15:19:24.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:19:24.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:19:24.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:19:24.754 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:19:24.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:19:24.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:19:25.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:19:26.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:19:26.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:19:26.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:19:26.024 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:19:26.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:19:26.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:19:26.139 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:26.137+0000 7f828f414640 -1 osd.0 167 log_to_monitors true 2024-08-19T15:19:26.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:19:27.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:19:27.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:19:27.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:19:27.410 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:19:27.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:19:27.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:19:27.769 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 171 up_thru 171 down_at 168 last_clean_interval [160,167) [v2:127.0.0.1:6802/4247458607,v1:127.0.0.1:6803/4247458607] [v2:127.0.0.1:6804/4247458607,v1:127.0.0.1:6805/4247458607] exists,up a9a4af16-d461-4c2f-ab0d-b99df37a8a3e 2024-08-19T15:19:27.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:19:27.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:19:27.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:19:27.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:19:27.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:19:27.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:19:27.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:19:27.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:19:27.770 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:19:27.770 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:19:27.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:19:27.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:19:27.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:19:27.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:19:27.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:19:27.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:19:27.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:19:27.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:19:27.863 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:19:28.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:19:28.129 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:19:28.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:19:28.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:19:28.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:19:28.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=734439407618 2024-08-19T15:19:28.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 734439407618 2024-08-19T15:19:28.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407618' 2024-08-19T15:19:28.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:19:28.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:19:28.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=708669603846 2024-08-19T15:19:28.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 708669603846 2024-08-19T15:19:28.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407618 1-708669603846' 2024-08-19T15:19:28.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:19:28.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-734439407618 2024-08-19T15:19:28.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:19:28.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:19:28.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-734439407618 2024-08-19T15:19:28.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:19:28.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=734439407618 2024-08-19T15:19:28.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 734439407618' 2024-08-19T15:19:28.337 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 734439407618 2024-08-19T15:19:28.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:19:28.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 734439407618 2024-08-19T15:19:28.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:19:29.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:19:29.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:19:29.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 734439407618 2024-08-19T15:19:29.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:19:30.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:19:30.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:19:31.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407618 -lt 734439407618 2024-08-19T15:19:31.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:19:31.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-708669603846 2024-08-19T15:19:31.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:19:31.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:19:31.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-708669603846 2024-08-19T15:19:31.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:19:31.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=708669603846 2024-08-19T15:19:31.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 708669603846' 2024-08-19T15:19:31.156 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 708669603846 2024-08-19T15:19:31.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:19:31.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 708669603846 -lt 708669603846 2024-08-19T15:19:31.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:19:31.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:19:31.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:19:31.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:19:31.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:19:31.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:19:31.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:19:31.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:19:31.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:19:31.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:19:31.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:19:32.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:19:32.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:19:32.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:19:32.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:19:32.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:19:32.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:19:32.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:19:32.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-08-19T15:19:32.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:19:32.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:19:32.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:19:32.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:19:32.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-08-19T15:19:32.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:19:32.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:19:32.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:19:32.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:19:32.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:19:32.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:19:32.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:19:32.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:19:32.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:19:32.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:19:32.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:19:32.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-08-19T15:19:32.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:19:32.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:19:32.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:19:32.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:19:32.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:19:32.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-08-19T15:19:33.552 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-08-19T15:19:33.552 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-08-19T15:19:34.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:19:34.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:19:34.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:19:34.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:19:34.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:19:34.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:19:34.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:19:34.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:19:34.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:19:34.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:19:34.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:19:34.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:19:34.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:19:34.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:19:34.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:19:34.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:19:34.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:19:34.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:19:34.092 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:19:34.092 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:19:34.092 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:19:34.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:19:34.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:19:34.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:19:34.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:19:34.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:19:34.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:19:34.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:19:34.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:19:34.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:19:34.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:19:34.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:19:34.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:19:34.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:19:34.093 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:19:34.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:19:34.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:19:34.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:19:34.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:19:34.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:19:34.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:19:34.118 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:34.117+0000 7f9250b23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:19:34.120 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:34.119+0000 7f9250b23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:19:34.121 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:34.120+0000 7f9250b23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:19:34.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:19:34.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:19:34.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:19:34.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:19:34.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:19:34.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:19:34.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:19:34.360 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:19:34.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:19:34.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:19:34.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:19:34.644 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:34.643+0000 7f9250b23640 -1 Falling back to public interface 2024-08-19T15:19:35.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:19:35.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:19:35.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:19:35.625 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:19:35.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:19:35.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:19:35.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:19:36.764 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:36.763+0000 7f9250b23640 -1 osd.1 173 log_to_monitors true 2024-08-19T15:19:36.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:19:36.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:19:36.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:19:36.923 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:19:36.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:19:36.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:19:37.309 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 177 up_thru 171 down_at 174 last_clean_interval [165,173) [v2:127.0.0.1:6810/1922070055,v1:127.0.0.1:6811/1922070055] [v2:127.0.0.1:6812/1922070055,v1:127.0.0.1:6813/1922070055] exists,up 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:19:37.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:19:37.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:19:37.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:19:37.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:19:37.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:19:37.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:19:37.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:19:37.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:19:37.310 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:19:37.310 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:19:37.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:19:37.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:19:37.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:19:37.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:19:37.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:19:37.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:19:37.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:19:37.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:19:37.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:19:37.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:19:37.685 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:19:37.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:19:37.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:19:37.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:19:37.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=734439407621 2024-08-19T15:19:37.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 734439407621 2024-08-19T15:19:37.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407621' 2024-08-19T15:19:37.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:19:37.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:19:38.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=760209211394 2024-08-19T15:19:38.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 760209211394 2024-08-19T15:19:38.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407621 1-760209211394' 2024-08-19T15:19:38.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:19:38.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-734439407621 2024-08-19T15:19:38.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:19:38.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:19:38.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-734439407621 2024-08-19T15:19:38.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:19:38.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=734439407621 2024-08-19T15:19:38.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 734439407621' 2024-08-19T15:19:38.089 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 734439407621 2024-08-19T15:19:38.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:19:38.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407620 -lt 734439407621 2024-08-19T15:19:38.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:19:39.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:19:39.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:19:39.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407620 -lt 734439407621 2024-08-19T15:19:39.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:19:40.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:19:40.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:19:40.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407621 -lt 734439407621 2024-08-19T15:19:40.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:19:40.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-760209211394 2024-08-19T15:19:40.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:19:40.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:19:40.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-760209211394 2024-08-19T15:19:40.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:19:40.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=760209211394 2024-08-19T15:19:40.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 760209211394' 2024-08-19T15:19:40.951 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 760209211394 2024-08-19T15:19:40.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:19:41.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 760209211394 -lt 760209211394 2024-08-19T15:19:41.221 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:19:41.221 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:19:41.222 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:19:41.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:19:41.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:19:41.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:19:41.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:19:41.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:19:41.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:19:41.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:19:41.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:19:41.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-08-19T15:19:41.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:19:41.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:19:41.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:19:42.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-08-19T15:19:42.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-08-19T15:19:42.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T15:19:42.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-08-19T15:19:42.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T15:19:42.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T15:19:42.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T15:19:42.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:19:42.265 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:19:42.265 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:19:42.265 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:19:42.265 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:19:42.265 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:19:42.265 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:19:42.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:19:42.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:19:42.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:19:42.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:19:42.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:19:42.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:19:42.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:19:42.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1137: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-08-19T15:19:42.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:19:42.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:19:42.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:19:42.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:19:42.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-08-19T15:19:42.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:19:42.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:19:42.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:19:42.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:19:42.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:19:42.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:19:42.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:19:42.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:19:42.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:19:42.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:19:43.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:19:43.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-08-19T15:19:43.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:19:43.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:19:43.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:19:43.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:19:43.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:19:43.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2024-08-19T15:19:44.134 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-08-19T15:19:44.134 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-08-19T15:19:44.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:19:44.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:19:44.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:19:44.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:19:44.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:19:44.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:19:44.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:19:44.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:19:44.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:19:44.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:19:44.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:19:44.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:19:44.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:19:44.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:19:44.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:19:44.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:19:44.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:19:44.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:19:44.654 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:19:44.654 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:19:44.654 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:19:44.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:19:44.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:19:44.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:19:44.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:19:44.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:19:44.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:19:44.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:19:44.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:19:44.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:19:44.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:19:44.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:19:44.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:19:44.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:19:44.656 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:19:44.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:19:44.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:19:44.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:19:44.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:19:44.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:19:44.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:19:44.683 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:44.682+0000 7fdb46b14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:19:44.685 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:44.684+0000 7fdb46b14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:19:44.687 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:44.686+0000 7fdb46b14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:19:44.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:19:44.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:19:44.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:19:44.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:19:44.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:19:44.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:19:44.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:19:44.934 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:19:44.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:19:44.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:19:45.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:19:45.217 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:45.216+0000 7fdb46b14640 -1 Falling back to public interface 2024-08-19T15:19:46.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:19:46.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:19:46.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:19:46.200 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:19:46.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:19:46.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:19:46.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:19:47.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:19:47.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:19:47.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:19:47.474 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:19:47.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:19:47.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:19:47.518 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:19:47.517+0000 7fdb46b14640 -1 osd.1 179 log_to_monitors true 2024-08-19T15:19:47.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:19:48.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:19:48.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:19:48.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:19:48.782 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:19:48.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:19:48.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:19:49.134 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 183 up_thru 183 down_at 180 last_clean_interval [177,179) [v2:127.0.0.1:6810/1362472308,v1:127.0.0.1:6811/1362472308] [v2:127.0.0.1:6812/1362472308,v1:127.0.0.1:6813/1362472308] exists,up 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:19:49.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:19:49.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:19:49.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:19:49.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:19:49.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:19:49.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:19:49.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:19:49.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:19:49.135 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:19:49.135 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:19:49.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:19:49.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:19:49.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:19:49.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:19:49.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:19:49.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:19:49.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:19:49.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:19:49.228 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:19:49.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:19:49.491 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:19:49.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:19:49.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:19:49.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:19:49.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=734439407624 2024-08-19T15:19:49.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 734439407624 2024-08-19T15:19:49.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407624' 2024-08-19T15:19:49.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:19:49.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:19:49.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=785979015170 2024-08-19T15:19:49.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 785979015170 2024-08-19T15:19:49.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407624 1-785979015170' 2024-08-19T15:19:49.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:19:49.695 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-734439407624 2024-08-19T15:19:49.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:19:49.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:19:49.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-734439407624 2024-08-19T15:19:49.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:19:49.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=734439407624 2024-08-19T15:19:49.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 734439407624' 2024-08-19T15:19:49.698 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 734439407624 2024-08-19T15:19:49.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:19:49.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407624 -lt 734439407624 2024-08-19T15:19:49.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:19:49.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-785979015170 2024-08-19T15:19:49.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:19:49.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:19:49.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-785979015170 2024-08-19T15:19:49.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:19:49.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=785979015170 2024-08-19T15:19:49.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 785979015170' 2024-08-19T15:19:49.968 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 785979015170 2024-08-19T15:19:49.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:19:50.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 785979015170 -lt 785979015170 2024-08-19T15:19:50.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:19:50.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:19:50.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:19:50.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:19:50.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:19:50.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:19:50.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:19:50.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:19:50.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:19:50.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:19:50.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:19:50.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:19:50.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:19:50.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:19:50.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:19:51.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:19:51.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:19:51.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:19:51.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-08-19T15:19:51.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-08-19T15:19:51.182 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 19 % 2 2024-08-19T15:19:51.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-08-19T15:19:51.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-08-19T15:19:51.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-08-19T15:19:51.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-08-19T15:19:51.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-08-19T15:19:51.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-08-19T15:19:51.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:19:51.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: local pg=3.0 2024-08-19T15:19:51.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2024-08-19T15:19:51.546 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-08-19T15:19:51.547 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-08-19T15:19:51.563 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-08-19T15:19:51.563 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-08-19T15:19:51.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1150: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-08-19T15:19:51.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-08-19T15:19:51.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-08-19T15:19:51.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-08-19T15:19:51.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-08-19T15:19:51.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-08-19T15:19:51.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-08-19T15:19:51.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-08-19T15:19:51.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-08-19T15:19:51.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-08-19T15:19:51.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-08-19T15:19:51.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-08-19T15:19:51.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-08-19T15:19:51.574 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-08-19T15:19:51.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-08-19T15:19:51.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-08-19T15:19:51.575 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-08-19T15:19:51.575 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-19T15:19:51.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-08-19T15:19:51.874 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-08-19T15:19:51.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-08-19T15:19:51.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-08-19T15:19:51.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-08-19T15:19:51.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-08-19T15:19:51.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-08-19T15:19:51.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-08-19T15:19:51.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-08-19T15:19:51.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-08-19T15:19:51.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-08-19T15:19:51.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-08-19T15:19:51.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-08-19T15:19:51.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-08-19T15:19:51.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-08-19T15:19:51.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-08-19T15:19:51.876 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-08-19T15:19:51.876 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T15:19:51.876 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T15:19:51.877 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:19:51.877 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:19:51.877 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:19:51.877 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.1.asok 2024-08-19T15:19:51.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.57280/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-08-19T15:19:51.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-08-19T15:19:51.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-08-19T15:19:51.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-08-19T15:19:51.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-08-19T15:19:51.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T15:19:51.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T15:19:51.964 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:19:51.964 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:19:51.964 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:19:51.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.1.asok 2024-08-19T15:19:51.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-08-19T15:19:51.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.57280/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-08-19T15:19:52.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1151: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-08-19T15:19:52.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-08-19T15:19:52.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-08-19T15:19:52.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-08-19T15:19:52.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-08-19T15:19:52.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-08-19T15:19:52.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-08-19T15:19:52.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-08-19T15:19:52.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-08-19T15:19:52.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-08-19T15:19:52.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-08-19T15:19:52.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-08-19T15:19:52.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-08-19T15:19:52.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-08-19T15:19:52.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-08-19T15:19:52.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-08-19T15:19:52.052 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-08-19T15:19:52.052 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-19T15:19:52.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-08-19T15:19:52.323 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-08-19T15:19:52.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-08-19T15:19:52.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-08-19T15:19:52.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-08-19T15:19:52.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-08-19T15:19:52.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-08-19T15:19:52.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-08-19T15:19:52.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-08-19T15:19:52.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-08-19T15:19:52.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-08-19T15:19:52.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-08-19T15:19:52.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-08-19T15:19:52.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-08-19T15:19:52.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-08-19T15:19:52.326 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-08-19T15:19:52.327 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-08-19T15:19:52.327 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T15:19:52.327 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T15:19:52.327 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:19:52.327 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:19:52.327 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:19:52.327 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.0.asok 2024-08-19T15:19:52.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.57280/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-08-19T15:19:52.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-08-19T15:19:52.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-08-19T15:19:52.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-08-19T15:19:52.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-08-19T15:19:52.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T15:19:52.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T15:19:52.413 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:19:52.413 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:19:52.413 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:19:52.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.0.asok 2024-08-19T15:19:52.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-08-19T15:19:52.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.57280/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-08-19T15:19:52.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-08-19T15:19:52.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-08-19T15:19:52.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-08-19T15:19:52.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-08-19T15:19:52.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-08-19T15:19:52.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-08-19T15:19:52.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-08-19T15:19:52.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-08-19T15:19:52.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-08-19T15:19:52.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-08-19T15:19:52.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-08-19T15:19:52.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-08-19T15:19:52.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-08-19T15:19:52.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-08-19T15:19:52.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-08-19T15:19:52.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-08-19T15:19:52.502 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-08-19T15:19:52.502 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-19T15:19:52.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-08-19T15:19:52.771 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-08-19T15:19:52.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-08-19T15:19:52.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-08-19T15:19:52.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-08-19T15:19:52.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-08-19T15:19:52.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-08-19T15:19:52.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-08-19T15:19:52.772 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-08-19T15:19:52.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-08-19T15:19:52.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-08-19T15:19:52.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-08-19T15:19:52.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-08-19T15:19:52.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-08-19T15:19:52.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-08-19T15:19:52.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-08-19T15:19:52.773 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-08-19T15:19:52.773 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T15:19:52.773 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T15:19:52.773 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:19:52.773 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:19:52.773 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:19:52.773 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.0.asok 2024-08-19T15:19:52.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.57280/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-08-19T15:19:52.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-08-19T15:19:52.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-08-19T15:19:52.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-08-19T15:19:52.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-08-19T15:19:52.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T15:19:52.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T15:19:52.856 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:19:52.856 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:19:52.856 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:19:52.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.0.asok 2024-08-19T15:19:52.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-08-19T15:19:52.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.57280/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-08-19T15:19:52.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1153: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-08-19T15:19:52.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-08-19T15:19:52.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-08-19T15:19:52.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-08-19T15:19:52.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-08-19T15:19:52.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-08-19T15:19:52.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-08-19T15:19:52.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-08-19T15:19:52.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-08-19T15:19:52.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-08-19T15:19:52.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-08-19T15:19:52.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-08-19T15:19:52.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-08-19T15:19:52.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-08-19T15:19:52.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-08-19T15:19:52.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-08-19T15:19:52.947 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-08-19T15:19:52.947 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-19T15:19:53.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-08-19T15:19:53.215 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-08-19T15:19:53.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-08-19T15:19:53.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-08-19T15:19:53.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-08-19T15:19:53.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-08-19T15:19:53.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-08-19T15:19:53.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-08-19T15:19:53.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-08-19T15:19:53.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-08-19T15:19:53.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-08-19T15:19:53.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-08-19T15:19:53.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-08-19T15:19:53.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-08-19T15:19:53.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-08-19T15:19:53.218 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-08-19T15:19:53.218 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-08-19T15:19:53.218 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T15:19:53.218 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T15:19:53.218 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:19:53.218 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:19:53.218 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:19:53.218 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.1.asok 2024-08-19T15:19:53.218 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.57280/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-08-19T15:19:53.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-08-19T15:19:53.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-08-19T15:19:53.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-08-19T15:19:53.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-08-19T15:19:53.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T15:19:53.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T15:19:53.304 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:19:53.304 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:19:53.304 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:19:53.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.1.asok 2024-08-19T15:19:53.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-08-19T15:19:53.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.57280/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-08-19T15:19:53.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1155: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2024-08-19T15:19:53.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-08-19T15:19:53.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-08-19T15:19:53.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:19:53.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:19:53.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:19:53.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:19:53.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-08-19T15:15:55.624899+0000 2024-08-19T15:19:53.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-08-19T15:19:53.906 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 3.0 on osd.1 to scrub 2024-08-19T15:19:53.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-08-19T15:15:55.624899+0000 2024-08-19T15:19:53.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-08-19T15:19:53.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:15:55.624899+0000 2024-08-19T15:19:53.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:19:53.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:19:53.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:19:53.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-08-19T15:19:53.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:19:53.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:19:53.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:19:53.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:19:54.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:15:55.624899+0000 '>' 2024-08-19T15:15:55.624899+0000 2024-08-19T15:19:54.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:19:55.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:19:55.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:19:55.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-08-19T15:19:55.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:19:55.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:19:55.184 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:19:55.184 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:19:55.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:15:55.624899+0000 '>' 2024-08-19T15:15:55.624899+0000 2024-08-19T15:19:55.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:19:56.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:19:56.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:19:56.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-08-19T15:19:56.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:19:56.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:19:56.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:19:56.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:19:56.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:15:55.624899+0000 '>' 2024-08-19T15:15:55.624899+0000 2024-08-19T15:19:56.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:19:57.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:19:57.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:19:57.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-08-19T15:19:57.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:19:57.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:19:57.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:19:57.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:19:58.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:19:54.574523+0000 '>' 2024-08-19T15:15:55.624899+0000 2024-08-19T15:19:58.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:19:58.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1157: TEST_corrupt_scrub_replicated: ERRORS=0 2024-08-19T15:19:58.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1158: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2024-08-19T15:19:58.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1159: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-08-19T15:19:58.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1160: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2024-08-19T15:19:58.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1161: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-08-19T15:19:58.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1162: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-08-19T15:19:58.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1163: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-08-19T15:19:58.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1164: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-08-19T15:19:58.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-08-19T15:19:58.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-08-19T15:19:58.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-08-19T15:19:58.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-08-19T15:19:58.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1169: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-08-19T15:19:58.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1170: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-08-19T15:19:58.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1171: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-08-19T15:19:58.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-08-19T15:19:58.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1173: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2024-08-19T15:19:58.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1174: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-08-19T15:19:58.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1175: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-08-19T15:19:58.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' 2024-08-19T15:19:58.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-08-19T15:19:58.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2024-08-19T15:19:58.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1179: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2024-08-19T15:19:58.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1180: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2024-08-19T15:19:58.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:19:58.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2024-08-19T15:19:58.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-08-19T15:19:58.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-08-19T15:19:58.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: test 1 = 1 2024-08-19T15:19:58.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-08-19T15:19:58.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-08-19T15:19:58.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-08-19T15:19:58.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-08-19T15:19:58.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: epoch=183 2024-08-19T15:19:58.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-08-19T15:19:58.271 INFO:tasks.workunit.client.0.smithi102.stderr: . as $in 2024-08-19T15:19:58.271 INFO:tasks.workunit.client.0.smithi102.stderr: | if type == "object" then 2024-08-19T15:19:58.271 INFO:tasks.workunit.client.0.smithi102.stderr: reduce keys[] as $key 2024-08-19T15:19:58.271 INFO:tasks.workunit.client.0.smithi102.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-08-19T15:19:58.271 INFO:tasks.workunit.client.0.smithi102.stderr: elif type == "array" then map( walk(f) ) | f 2024-08-19T15:19:58.271 INFO:tasks.workunit.client.0.smithi102.stderr: else f 2024-08-19T15:19:58.271 INFO:tasks.workunit.client.0.smithi102.stderr: end; 2024-08-19T15:19:58.271 INFO:tasks.workunit.client.0.smithi102.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-08-19T15:19:58.271 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-08-19T15:19:58.271 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-08-19T15:19:58.271 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.version) else . end) 2024-08-19T15:19:58.271 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-08-19T15:19:58.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-08-19T15:19:58.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-08-19T15:19:58.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-08-19T15:19:58.294 INFO:tasks.workunit.client.0.smithi102.stderr: . as $in 2024-08-19T15:19:58.294 INFO:tasks.workunit.client.0.smithi102.stderr: | if type == "object" then 2024-08-19T15:19:58.294 INFO:tasks.workunit.client.0.smithi102.stderr: reduce keys[] as $key 2024-08-19T15:19:58.294 INFO:tasks.workunit.client.0.smithi102.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-08-19T15:19:58.294 INFO:tasks.workunit.client.0.smithi102.stderr: elif type == "array" then map( walk(f) ) | f 2024-08-19T15:19:58.295 INFO:tasks.workunit.client.0.smithi102.stderr: else f 2024-08-19T15:19:58.295 INFO:tasks.workunit.client.0.smithi102.stderr: end; 2024-08-19T15:19:58.295 INFO:tasks.workunit.client.0.smithi102.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-08-19T15:19:58.295 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-08-19T15:19:58.295 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-08-19T15:19:58.295 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.version) else . end) 2024-08-19T15:19:58.295 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-08-19T15:19:58.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-08-19T15:19:58.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-08-19T15:19:58.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2083: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-08-19T15:19:58.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-08-19T15:19:58.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2084: TEST_corrupt_scrub_replicated: test no = yes 2024-08-19T15:19:58.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2089: TEST_corrupt_scrub_replicated: test '' = yes 2024-08-19T15:19:58.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2094: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-08-19T15:19:58.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2096: TEST_corrupt_scrub_replicated: echo -n ZZZ 2024-08-19T15:19:58.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2097: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-08-19T15:19:58.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-08-19T15:19:58.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:19:58.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:19:58.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:19:58.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:19:58.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-08-19T15:19:58.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:19:58.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:19:58.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:19:58.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:19:58.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:19:58.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:19:58.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:19:58.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:19:58.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:19:58.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:19:58.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:19:58.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-08-19T15:19:58.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:19:58.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:19:58.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:19:58.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:19:58.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:19:58.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-08-19T15:19:59.664 INFO:tasks.workunit.client.0.smithi102.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-08-19T15:20:00.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:20:00.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:20:00.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:20:00.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:20:00.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:20:00.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:20:00.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:20:00.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:20:00.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:20:00.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:20:00.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:20:00.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:20:00.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:20:00.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:20:00.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:20:00.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:20:00.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:20:00.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:20:00.183 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:20:00.183 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:20:00.183 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:20:00.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:20:00.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:20:00.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:20:00.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:20:00.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:20:00.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:20:00.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:20:00.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:20:00.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:20:00.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:20:00.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:20:00.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:20:00.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:20:00.185 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:20:00.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:20:00.185 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:20:00.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:20:00.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:20:00.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:20:00.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:20:00.212 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:00.211+0000 7f8ec6417640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:20:00.215 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:00.214+0000 7f8ec6417640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:20:00.216 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:00.215+0000 7f8ec6417640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:20:00.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:20:00.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:20:00.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:20:00.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:20:00.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:20:00.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:20:00.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:20:00.465 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:20:00.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:20:00.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:20:00.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:20:00.736 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:00.735+0000 7f8ec6417640 -1 Falling back to public interface 2024-08-19T15:20:01.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:20:01.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:20:01.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:20:01.731 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:20:01.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:20:01.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:20:02.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:20:03.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:20:03.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:20:03.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:20:03.006 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:20:03.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:20:03.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:20:03.040 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:03.039+0000 7f8ec6417640 -1 osd.0 185 log_to_monitors true 2024-08-19T15:20:03.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:20:04.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:20:04.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:20:04.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:20:04.283 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:20:04.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:20:04.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:20:04.561 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 189 up_thru 189 down_at 186 last_clean_interval [171,185) [v2:127.0.0.1:6802/2112136694,v1:127.0.0.1:6803/2112136694] [v2:127.0.0.1:6804/2112136694,v1:127.0.0.1:6805/2112136694] exists,up a9a4af16-d461-4c2f-ab0d-b99df37a8a3e 2024-08-19T15:20:04.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:20:04.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:20:04.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:20:04.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:20:04.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:20:04.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:20:04.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:20:04.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:20:04.563 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:20:04.563 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:20:04.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:20:04.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:20:04.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:20:04.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:20:04.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:20:04.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:20:04.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:20:04.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:20:04.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:20:04.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:20:04.921 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:20:04.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:20:04.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:20:04.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:20:05.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=811748818946 2024-08-19T15:20:05.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 811748818946 2024-08-19T15:20:05.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818946' 2024-08-19T15:20:05.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:20:05.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:20:05.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=785979015174 2024-08-19T15:20:05.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 785979015174 2024-08-19T15:20:05.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818946 1-785979015174' 2024-08-19T15:20:05.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:20:05.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-811748818946 2024-08-19T15:20:05.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:20:05.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:20:05.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-811748818946 2024-08-19T15:20:05.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:20:05.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=811748818946 2024-08-19T15:20:05.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 811748818946' 2024-08-19T15:20:05.128 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 811748818946 2024-08-19T15:20:05.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:20:05.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 811748818946 2024-08-19T15:20:05.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:20:06.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:20:06.398 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:20:06.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 811748818946 -lt 811748818946 2024-08-19T15:20:06.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:20:06.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-785979015174 2024-08-19T15:20:06.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:20:06.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:20:06.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-785979015174 2024-08-19T15:20:06.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:20:06.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=785979015174 2024-08-19T15:20:06.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 785979015174' 2024-08-19T15:20:06.674 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 785979015174 2024-08-19T15:20:06.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:20:06.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 785979015174 -lt 785979015174 2024-08-19T15:20:06.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:20:06.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:20:06.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:20:07.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:20:07.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:20:07.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:20:07.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:20:07.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:20:07.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:20:07.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:20:07.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:20:07.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:20:07.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:20:07.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:20:07.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:20:07.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:20:07.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:20:07.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:20:07.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2100: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-08-19T15:20:07.873 INFO:tasks.workunit.client.0.smithi102.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-08-19T15:20:07.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2102: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-08-19T15:20:07.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2103: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-08-19T15:20:07.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:20:07.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:20:07.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:20:07.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:20:07.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-08-19T15:20:07.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:20:07.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:20:07.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:20:07.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:20:07.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:20:07.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:20:07.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:20:07.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:20:07.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:20:07.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:20:08.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:20:08.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-08-19T15:20:08.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:20:08.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:20:08.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:20:08.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:20:08.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:20:08.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2024-08-19T15:20:08.864 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-08-19T15:20:08.864 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-08-19T15:20:09.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:20:09.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:20:09.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:20:09.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:20:09.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:20:09.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:20:09.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:20:09.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:20:09.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:20:09.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:20:09.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:20:09.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:20:09.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:20:09.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:20:09.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:20:09.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:20:09.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:20:09.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:20:09.383 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:20:09.383 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:20:09.383 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:20:09.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:20:09.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:20:09.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:20:09.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:20:09.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:20:09.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:20:09.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:20:09.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:20:09.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:20:09.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:20:09.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:20:09.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:20:09.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:20:09.386 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:20:09.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:20:09.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:20:09.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:20:09.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:20:09.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:20:09.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:20:09.411 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:09.410+0000 7fd86222e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:20:09.413 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:09.412+0000 7fd86222e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:20:09.415 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:09.414+0000 7fd86222e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:20:09.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:20:09.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:20:09.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:20:09.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:20:09.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:20:09.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:20:09.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:20:09.656 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:20:09.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:20:09.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:20:09.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:20:09.943 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:09.942+0000 7fd86222e640 -1 Falling back to public interface 2024-08-19T15:20:10.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:20:10.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:20:10.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:20:10.925 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:20:10.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:20:10.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:20:11.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:20:12.147 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:12.145+0000 7fd86222e640 -1 osd.1 191 log_to_monitors true 2024-08-19T15:20:12.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:20:12.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:20:12.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:20:12.197 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:20:12.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:20:12.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:20:12.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:20:13.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:20:13.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:20:13.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:20:13.547 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:20:13.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:20:13.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:20:13.826 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 195 up_thru 195 down_at 192 last_clean_interval [183,191) [v2:127.0.0.1:6810/474317073,v1:127.0.0.1:6811/474317073] [v2:127.0.0.1:6812/474317073,v1:127.0.0.1:6813/474317073] exists,up 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:20:13.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:20:13.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:20:13.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:20:13.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:20:13.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:20:13.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:20:13.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:20:13.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:20:13.826 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:20:13.827 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:20:13.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:20:13.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:20:13.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:20:13.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:20:13.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:20:13.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:20:13.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:20:13.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:20:13.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:20:14.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:20:14.185 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:20:14.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:20:14.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:20:14.185 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:20:14.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=811748818949 2024-08-19T15:20:14.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 811748818949 2024-08-19T15:20:14.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818949' 2024-08-19T15:20:14.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:20:14.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:20:14.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=837518622722 2024-08-19T15:20:14.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 837518622722 2024-08-19T15:20:14.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818949 1-837518622722' 2024-08-19T15:20:14.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:20:14.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-811748818949 2024-08-19T15:20:14.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:20:14.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:20:14.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-811748818949 2024-08-19T15:20:14.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:20:14.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=811748818949 2024-08-19T15:20:14.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 811748818949' 2024-08-19T15:20:14.393 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 811748818949 2024-08-19T15:20:14.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:20:14.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 811748818948 -lt 811748818949 2024-08-19T15:20:14.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:20:15.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:20:15.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:20:15.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 811748818948 -lt 811748818949 2024-08-19T15:20:15.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:20:16.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:20:16.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:20:17.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 811748818949 -lt 811748818949 2024-08-19T15:20:17.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:20:17.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-837518622722 2024-08-19T15:20:17.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:20:17.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:20:17.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-837518622722 2024-08-19T15:20:17.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:20:17.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=837518622722 2024-08-19T15:20:17.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 837518622722' 2024-08-19T15:20:17.207 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 837518622722 2024-08-19T15:20:17.208 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:20:17.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 837518622722 -lt 837518622722 2024-08-19T15:20:17.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:20:17.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:20:17.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:20:17.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:20:17.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:20:17.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:20:17.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:20:17.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:20:17.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:20:17.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:20:17.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:20:18.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:20:18.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:20:18.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:20:18.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:20:18.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:20:18.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:20:18.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:20:18.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2104: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2024-08-19T15:20:18.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2105: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-08-19T15:20:18.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:20:18.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:20:18.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:20:18.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:20:18.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-08-19T15:20:18.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:20:18.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:20:18.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:20:18.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:20:18.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:20:18.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:20:18.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:20:18.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:20:18.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:20:18.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:20:18.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:20:18.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-08-19T15:20:18.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:20:18.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:20:18.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:20:18.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:20:18.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:20:18.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-08-19T15:20:20.071 INFO:tasks.workunit.client.0.smithi102.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-08-19T15:20:20.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:20:20.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:20:20.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:20:20.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:20:20.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:20:20.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:20:20.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:20:20.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:20:20.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:20:20.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:20:20.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:20:20.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:20:20.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:20:20.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:20:20.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:20:20.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:20:20.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:20:20.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:20:20.597 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:20:20.597 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:20:20.597 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:20:20.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:20:20.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:20:20.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:20:20.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:20:20.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:20:20.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:20:20.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:20:20.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:20:20.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:20:20.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:20:20.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:20:20.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:20:20.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:20:20.599 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:20:20.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:20:20.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:20:20.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:20:20.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:20:20.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:20:20.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:20:20.625 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:20.624+0000 7f921f72a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:20:20.627 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:20.626+0000 7f921f72a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:20:20.628 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:20.627+0000 7f921f72a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:20:20.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:20:20.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:20:20.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:20:20.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:20:20.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:20:20.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:20:20.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:20:20.872 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:20:20.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:20:20.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:20:21.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:20:21.154 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:21.153+0000 7f921f72a640 -1 Falling back to public interface 2024-08-19T15:20:22.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:20:22.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:20:22.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:20:22.139 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:20:22.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:20:22.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:20:22.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:20:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:20:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:20:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:20:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:20:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:20:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:20:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:23.517+0000 7f921f72a640 -1 osd.0 197 log_to_monitors true 2024-08-19T15:20:23.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:20:24.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:20:24.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:20:24.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:20:24.689 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:20:24.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:20:24.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:20:25.038 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 201 up_thru 201 down_at 198 last_clean_interval [189,197) [v2:127.0.0.1:6802/1187590778,v1:127.0.0.1:6803/1187590778] [v2:127.0.0.1:6804/1187590778,v1:127.0.0.1:6805/1187590778] exists,up a9a4af16-d461-4c2f-ab0d-b99df37a8a3e 2024-08-19T15:20:25.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:20:25.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:20:25.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:20:25.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:20:25.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:20:25.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:20:25.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:20:25.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:20:25.040 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:20:25.040 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:20:25.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:20:25.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:20:25.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:20:25.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:20:25.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:20:25.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:20:25.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:20:25.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:20:25.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:20:25.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:20:25.398 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:20:25.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:20:25.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:20:25.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:20:25.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=863288426498 2024-08-19T15:20:25.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 863288426498 2024-08-19T15:20:25.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426498' 2024-08-19T15:20:25.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:20:25.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:20:25.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=837518622725 2024-08-19T15:20:25.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 837518622725 2024-08-19T15:20:25.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426498 1-837518622725' 2024-08-19T15:20:25.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:20:25.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-863288426498 2024-08-19T15:20:25.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:20:25.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:20:25.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-863288426498 2024-08-19T15:20:25.604 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:20:25.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=863288426498 2024-08-19T15:20:25.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 863288426498' 2024-08-19T15:20:25.605 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 863288426498 2024-08-19T15:20:25.605 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:20:25.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 863288426498 2024-08-19T15:20:25.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:20:26.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:20:26.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:20:27.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426498 -lt 863288426498 2024-08-19T15:20:27.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:20:27.150 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-837518622725 2024-08-19T15:20:27.150 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:20:27.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:20:27.151 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-837518622725 2024-08-19T15:20:27.151 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:20:27.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=837518622725 2024-08-19T15:20:27.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 837518622725' 2024-08-19T15:20:27.152 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 837518622725 2024-08-19T15:20:27.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:20:27.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 837518622725 -lt 837518622725 2024-08-19T15:20:27.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:20:27.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:20:27.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:20:27.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:20:27.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:20:27.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:20:27.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:20:27.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:20:27.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:20:27.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:20:27.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:20:28.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:20:28.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:20:28.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:20:28.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:20:28.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:20:28.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:20:28.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:20:28.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-08-19T15:20:28.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:20:28.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:20:28.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:20:28.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:20:28.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-08-19T15:20:28.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:20:28.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:20:28.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:20:28.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:20:28.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:20:28.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:20:28.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:20:28.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:20:28.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:20:28.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:20:28.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:20:28.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-08-19T15:20:28.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:20:28.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:20:28.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:20:28.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:20:28.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:20:28.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-08-19T15:20:29.524 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-08-19T15:20:29.525 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-08-19T15:20:30.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:20:30.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:20:30.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:20:30.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:20:30.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:20:30.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:20:30.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:20:30.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:20:30.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:20:30.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:20:30.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:20:30.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:20:30.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:20:30.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:20:30.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:20:30.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:20:30.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:20:30.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:20:30.054 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:20:30.054 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:20:30.054 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:20:30.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:20:30.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:20:30.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:20:30.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:20:30.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:20:30.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:20:30.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:20:30.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:20:30.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:20:30.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:20:30.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:20:30.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:20:30.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:20:30.055 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:20:30.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:20:30.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:20:30.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:20:30.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:20:30.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:20:30.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:20:30.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:30.076+0000 7f34a0f23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:20:30.079 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:30.078+0000 7f34a0f23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:20:30.080 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:30.079+0000 7f34a0f23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:20:30.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:20:30.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:20:30.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:20:30.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:20:30.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:20:30.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:20:30.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:20:30.323 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:20:30.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:20:30.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:20:30.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:20:30.609 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:30.608+0000 7f34a0f23640 -1 Falling back to public interface 2024-08-19T15:20:31.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:20:31.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:20:31.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:20:31.591 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:20:31.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:20:31.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:20:31.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:20:32.707 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:20:32.706+0000 7f34a0f23640 -1 osd.1 203 log_to_monitors true 2024-08-19T15:20:32.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:20:32.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:20:32.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:20:32.864 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:20:32.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:20:32.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:20:33.287 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 207 up_thru 201 down_at 204 last_clean_interval [195,203) [v2:127.0.0.1:6810/2903376154,v1:127.0.0.1:6811/2903376154] [v2:127.0.0.1:6812/2903376154,v1:127.0.0.1:6813/2903376154] exists,up 1a4b64bb-c6cc-4bea-bbf3-8d6731dc755b 2024-08-19T15:20:33.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:20:33.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:20:33.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:20:33.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:20:33.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:20:33.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:20:33.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:20:33.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:20:33.288 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:20:33.288 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:20:33.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:20:33.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:20:33.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:20:33.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:20:33.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:20:33.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:20:33.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:20:33.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:20:33.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:20:33.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:20:33.763 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:20:33.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:20:33.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:20:33.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:20:33.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=863288426501 2024-08-19T15:20:33.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 863288426501 2024-08-19T15:20:33.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426501' 2024-08-19T15:20:33.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:20:33.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:20:34.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=889058230274 2024-08-19T15:20:34.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 889058230274 2024-08-19T15:20:34.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426501 1-889058230274' 2024-08-19T15:20:34.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:20:34.207 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-863288426501 2024-08-19T15:20:34.207 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:20:34.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:20:34.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-863288426501 2024-08-19T15:20:34.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:20:34.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=863288426501 2024-08-19T15:20:34.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 863288426501' 2024-08-19T15:20:34.210 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 863288426501 2024-08-19T15:20:34.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:20:34.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426500 -lt 863288426501 2024-08-19T15:20:34.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:20:35.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:20:35.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:20:35.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426500 -lt 863288426501 2024-08-19T15:20:35.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:20:36.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:20:36.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:20:37.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426501 -lt 863288426501 2024-08-19T15:20:37.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:20:37.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-889058230274 2024-08-19T15:20:37.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:20:37.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:20:37.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-889058230274 2024-08-19T15:20:37.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:20:37.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=889058230274 2024-08-19T15:20:37.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 889058230274' 2024-08-19T15:20:37.064 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 889058230274 2024-08-19T15:20:37.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:20:37.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 889058230274 -lt 889058230274 2024-08-19T15:20:37.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:20:37.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:20:37.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:20:37.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:20:37.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:20:37.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:20:37.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:20:37.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:20:37.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:20:37.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:20:37.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:20:37.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-08-19T15:20:37.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:20:37.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:20:37.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:20:38.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-08-19T15:20:38.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-08-19T15:20:38.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T15:20:38.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-08-19T15:20:38.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T15:20:38.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T15:20:38.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T15:20:38.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:20:38.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:20:38.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:20:38.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:20:38.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:20:38.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:20:38.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:20:38.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-08-19T15:20:38.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:20:38.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:20:38.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:20:38.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-08-19T15:20:38.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-08-19T15:20:38.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T15:20:38.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T15:20:38.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T15:20:38.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T15:20:38.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T15:20:38.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T15:20:38.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T15:20:39.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T15:20:39.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T15:20:39.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-19T15:20:39.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T15:20:39.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-19T15:20:39.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T15:20:39.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:20:39.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:20:39.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:20:39.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:20:39.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:20:39.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:20:39.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:20:39.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-08-19T15:20:39.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:20:39.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:20:39.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:20:40.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-08-19T15:20:40.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-08-19T15:20:40.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T15:20:40.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T15:20:40.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T15:20:40.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T15:20:40.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T15:20:40.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T15:20:40.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T15:20:40.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T15:20:40.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T15:20:40.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-08-19T15:20:40.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T15:20:40.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-19T15:20:40.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T15:20:40.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:20:40.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:20:40.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:20:40.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:20:40.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:20:40.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:20:40.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:20:41.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:20:41.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:20:41.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:20:41.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:20:41.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:20:41.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:20:41.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:20:41.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2107: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2024-08-19T15:20:41.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2109: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-08-19T15:20:41.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-08-19T15:20:41.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-08-19T15:20:41.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-08-19T15:20:41.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-08-19T15:20:41.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-08-19T15:20:41.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-08-19T15:20:41.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-08-19T15:20:41.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-08-19T15:20:41.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-08-19T15:20:41.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-08-19T15:20:41.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-08-19T15:20:41.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-08-19T15:20:41.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-08-19T15:20:41.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-08-19T15:20:41.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-08-19T15:20:41.426 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-08-19T15:20:41.426 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-19T15:20:41.695 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-08-19T15:20:41.696 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-08-19T15:20:41.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-08-19T15:20:41.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-08-19T15:20:41.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-08-19T15:20:41.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-08-19T15:20:41.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-08-19T15:20:41.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-08-19T15:20:41.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-08-19T15:20:41.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-08-19T15:20:41.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-08-19T15:20:41.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-08-19T15:20:41.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-08-19T15:20:41.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-08-19T15:20:41.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-08-19T15:20:41.698 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-08-19T15:20:41.698 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-08-19T15:20:41.698 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T15:20:41.699 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T15:20:41.699 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:20:41.699 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:20:41.699 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:20:41.699 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.1.asok 2024-08-19T15:20:41.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.57280/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-08-19T15:20:41.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-08-19T15:20:41.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-08-19T15:20:41.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-08-19T15:20:41.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-08-19T15:20:41.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T15:20:41.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T15:20:41.785 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:20:41.785 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:20:41.785 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:20:41.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.1.asok 2024-08-19T15:20:41.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-08-19T15:20:41.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.57280/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-08-19T15:20:41.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2110: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-08-19T15:20:41.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-08-19T15:20:41.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-08-19T15:20:41.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-08-19T15:20:41.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-08-19T15:20:41.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-08-19T15:20:41.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-08-19T15:20:41.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-08-19T15:20:41.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-08-19T15:20:41.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-08-19T15:20:41.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-08-19T15:20:41.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-08-19T15:20:41.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-08-19T15:20:41.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-08-19T15:20:41.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-08-19T15:20:41.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-08-19T15:20:41.874 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-08-19T15:20:41.875 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-19T15:20:42.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-08-19T15:20:42.144 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-08-19T15:20:42.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-08-19T15:20:42.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-08-19T15:20:42.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-08-19T15:20:42.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-08-19T15:20:42.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-08-19T15:20:42.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-08-19T15:20:42.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-08-19T15:20:42.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-08-19T15:20:42.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-08-19T15:20:42.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-08-19T15:20:42.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-08-19T15:20:42.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-08-19T15:20:42.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-08-19T15:20:42.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-08-19T15:20:42.147 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-08-19T15:20:42.147 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T15:20:42.147 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T15:20:42.147 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:20:42.147 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:20:42.147 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:20:42.147 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.0.asok 2024-08-19T15:20:42.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.57280/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-08-19T15:20:42.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-08-19T15:20:42.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-08-19T15:20:42.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-08-19T15:20:42.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-08-19T15:20:42.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T15:20:42.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T15:20:42.234 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:20:42.234 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:20:42.234 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:20:42.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.0.asok 2024-08-19T15:20:42.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-08-19T15:20:42.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.57280/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-08-19T15:20:42.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2111: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-08-19T15:20:42.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-08-19T15:20:42.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-08-19T15:20:42.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-08-19T15:20:42.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-08-19T15:20:42.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-08-19T15:20:42.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-08-19T15:20:42.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-08-19T15:20:42.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-08-19T15:20:42.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-08-19T15:20:42.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-08-19T15:20:42.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-08-19T15:20:42.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-08-19T15:20:42.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-08-19T15:20:42.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-08-19T15:20:42.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-08-19T15:20:42.323 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-08-19T15:20:42.323 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-19T15:20:42.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-08-19T15:20:42.589 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-08-19T15:20:42.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-08-19T15:20:42.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-08-19T15:20:42.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-08-19T15:20:42.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-08-19T15:20:42.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-08-19T15:20:42.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-08-19T15:20:42.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-08-19T15:20:42.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-08-19T15:20:42.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-08-19T15:20:42.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-08-19T15:20:42.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-08-19T15:20:42.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-08-19T15:20:42.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-08-19T15:20:42.591 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-08-19T15:20:42.591 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-08-19T15:20:42.591 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T15:20:42.591 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T15:20:42.592 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:20:42.592 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:20:42.592 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:20:42.593 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.0.asok 2024-08-19T15:20:42.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.57280/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-08-19T15:20:42.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-08-19T15:20:42.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-08-19T15:20:42.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-08-19T15:20:42.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-08-19T15:20:42.678 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T15:20:42.678 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T15:20:42.678 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:20:42.678 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:20:42.678 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:20:42.678 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.0.asok 2024-08-19T15:20:42.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-08-19T15:20:42.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.57280/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-08-19T15:20:42.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2112: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-08-19T15:20:42.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-08-19T15:20:42.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-08-19T15:20:42.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-08-19T15:20:42.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-08-19T15:20:42.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-08-19T15:20:42.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-08-19T15:20:42.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-08-19T15:20:42.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-08-19T15:20:42.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-08-19T15:20:42.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-08-19T15:20:42.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-08-19T15:20:42.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-08-19T15:20:42.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-08-19T15:20:42.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-08-19T15:20:42.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-08-19T15:20:42.767 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-08-19T15:20:42.767 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-19T15:20:43.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-08-19T15:20:43.033 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-08-19T15:20:43.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-08-19T15:20:43.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-08-19T15:20:43.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-08-19T15:20:43.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-08-19T15:20:43.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-08-19T15:20:43.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-08-19T15:20:43.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-08-19T15:20:43.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-08-19T15:20:43.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-08-19T15:20:43.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-08-19T15:20:43.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-08-19T15:20:43.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-08-19T15:20:43.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-08-19T15:20:43.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-08-19T15:20:43.036 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-08-19T15:20:43.036 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T15:20:43.036 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T15:20:43.036 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:20:43.036 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:20:43.036 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:20:43.036 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.1.asok 2024-08-19T15:20:43.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.57280/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-08-19T15:20:43.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-08-19T15:20:43.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-08-19T15:20:43.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-08-19T15:20:43.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-08-19T15:20:43.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T15:20:43.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T15:20:43.122 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:20:43.122 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:20:43.122 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:20:43.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-osd.1.asok 2024-08-19T15:20:43.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-08-19T15:20:43.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.57280/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-08-19T15:20:43.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2115: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2024-08-19T15:20:43.302 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-08-19T15:20:43.303 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0: osd_max_object_size = '' 2024-08-19T15:20:43.319 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-08-19T15:20:43.319 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1: osd_max_object_size = '' 2024-08-19T15:20:43.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2117: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-08-19T15:20:43.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-08-19T15:20:43.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:20:43.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:20:43.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:20:43.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:20:43.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:20:43.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-08-19T15:15:55.624899+0000 2024-08-19T15:20:43.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-08-19T15:20:43.844 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-08-19T15:20:43.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-08-19T15:15:55.624899+0000 last_deep_scrub_stamp 2024-08-19T15:20:43.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-08-19T15:20:43.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:15:55.624899+0000 2024-08-19T15:20:43.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-08-19T15:20:43.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:20:43.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:20:43.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:20:43.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:20:43.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:20:43.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:20:43.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:20:44.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:15:55.624899+0000 '>' 2024-08-19T15:15:55.624899+0000 2024-08-19T15:20:44.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:20:45.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:20:45.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:20:45.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:20:45.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:20:45.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:20:45.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:20:45.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:20:45.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:15:55.624899+0000 '>' 2024-08-19T15:15:55.624899+0000 2024-08-19T15:20:45.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:20:46.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:20:46.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:20:46.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:20:46.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:20:46.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:20:46.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:20:46.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:20:46.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:15:55.624899+0000 '>' 2024-08-19T15:15:55.624899+0000 2024-08-19T15:20:46.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:20:47.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:20:47.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:20:47.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:20:47.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:20:47.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:20:47.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:20:47.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:20:47.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:15:55.624899+0000 '>' 2024-08-19T15:15:55.624899+0000 2024-08-19T15:20:47.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:20:48.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:20:48.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:20:48.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:20:48.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:20:48.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:20:48.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:20:48.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:20:49.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:15:55.624899+0000 '>' 2024-08-19T15:15:55.624899+0000 2024-08-19T15:20:49.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:20:50.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:20:50.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:20:50.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-08-19T15:20:50.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:20:50.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:20:50.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:20:50.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-08-19T15:20:50.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:20:44.677723+0000 '>' 2024-08-19T15:15:55.624899+0000 2024-08-19T15:20:50.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:20:50.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2119: TEST_corrupt_scrub_replicated: err_strings=() 2024-08-19T15:20:50.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2120: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-08-19T15:20:50.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2121: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2024-08-19T15:20:50.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2122: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2024-08-19T15:20:50.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2123: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2024-08-19T15:20:50.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2124: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-08-19T15:20:50.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2125: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-08-19T15:20:50.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2126: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-08-19T15:20:50.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2127: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-08-19T15:20:50.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2128: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2024-08-19T15:20:50.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2129: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-08-19T15:20:50.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2130: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-08-19T15:20:50.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2131: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2024-08-19T15:20:50.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2132: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2024-08-19T15:20:50.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2133: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2024-08-19T15:20:50.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2134: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2024-08-19T15:20:50.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2135: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-08-19T15:20:50.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2136: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-08-19T15:20:50.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2137: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2024-08-19T15:20:50.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2138: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-08-19T15:20:50.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2139: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-08-19T15:20:50.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2140: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-08-19T15:20:50.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2141: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-08-19T15:20:50.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2142: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-08-19T15:20:50.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2143: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-08-19T15:20:50.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2144: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2024-08-19T15:20:50.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2145: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-08-19T15:20:50.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2146: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2024-08-19T15:20:50.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2147: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2024-08-19T15:20:50.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2148: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-08-19T15:20:50.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2149: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2024-08-19T15:20:50.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2150: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2024-08-19T15:20:50.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2151: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2024-08-19T15:20:50.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2152: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2024-08-19T15:20:50.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2153: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-08-19T15:20:50.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2154: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-08-19T15:20:50.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2155: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' 2024-08-19T15:20:50.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2156: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-08-19T15:20:50.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2157: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2024-08-19T15:20:50.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2158: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2024-08-19T15:20:50.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-08-19T15:20:50.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2024-08-19T15:20:50.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2169: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-08-19T15:20:50.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-08-19T15:20:50.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: test 1 = 1 2024-08-19T15:20:50.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-08-19T15:20:50.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-08-19T15:20:50.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-08-19T15:20:50.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-08-19T15:20:50.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: epoch=207 2024-08-19T15:20:50.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-08-19T15:20:50.957 INFO:tasks.workunit.client.0.smithi102.stderr: . as $in 2024-08-19T15:20:50.957 INFO:tasks.workunit.client.0.smithi102.stderr: | if type == "object" then 2024-08-19T15:20:50.957 INFO:tasks.workunit.client.0.smithi102.stderr: reduce keys[] as $key 2024-08-19T15:20:50.957 INFO:tasks.workunit.client.0.smithi102.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-08-19T15:20:50.957 INFO:tasks.workunit.client.0.smithi102.stderr: elif type == "array" then map( walk(f) ) | f 2024-08-19T15:20:50.957 INFO:tasks.workunit.client.0.smithi102.stderr: else f 2024-08-19T15:20:50.957 INFO:tasks.workunit.client.0.smithi102.stderr: end; 2024-08-19T15:20:50.957 INFO:tasks.workunit.client.0.smithi102.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-08-19T15:20:50.957 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-08-19T15:20:50.958 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-08-19T15:20:50.958 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.version) else . end) 2024-08-19T15:20:50.958 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-08-19T15:20:50.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-08-19T15:20:50.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-08-19T15:20:50.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-08-19T15:20:50.981 INFO:tasks.workunit.client.0.smithi102.stderr: . as $in 2024-08-19T15:20:50.981 INFO:tasks.workunit.client.0.smithi102.stderr: | if type == "object" then 2024-08-19T15:20:50.981 INFO:tasks.workunit.client.0.smithi102.stderr: reduce keys[] as $key 2024-08-19T15:20:50.981 INFO:tasks.workunit.client.0.smithi102.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-08-19T15:20:50.981 INFO:tasks.workunit.client.0.smithi102.stderr: elif type == "array" then map( walk(f) ) | f 2024-08-19T15:20:50.981 INFO:tasks.workunit.client.0.smithi102.stderr: else f 2024-08-19T15:20:50.981 INFO:tasks.workunit.client.0.smithi102.stderr: end; 2024-08-19T15:20:50.981 INFO:tasks.workunit.client.0.smithi102.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-08-19T15:20:50.981 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-08-19T15:20:50.981 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-08-19T15:20:50.982 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.version) else . end) 2024-08-19T15:20:50.982 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-08-19T15:20:50.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-08-19T15:20:50.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-08-19T15:20:51.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3559: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-08-19T15:20:51.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-08-19T15:20:51.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3560: TEST_corrupt_scrub_replicated: test no = yes 2024-08-19T15:20:51.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3565: TEST_corrupt_scrub_replicated: test '' = yes 2024-08-19T15:20:51.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3570: TEST_corrupt_scrub_replicated: repair 3.0 2024-08-19T15:20:51.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=3.0 2024-08-19T15:20:51.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 3.0 2024-08-19T15:20:51.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:20:51.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:20:51.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:20:51.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:20:51.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T15:20:44.677723+0000 2024-08-19T15:20:51.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 3.0 2024-08-19T15:20:51.517 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 3.0 on osd.1 to repair 2024-08-19T15:20:51.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 3.0 2024-08-19T15:20:44.677723+0000 2024-08-19T15:20:51.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-08-19T15:20:51.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:20:44.677723+0000 2024-08-19T15:20:51.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:20:51.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:20:51.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:20:51.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-08-19T15:20:51.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:20:51.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:20:51.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:20:51.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:20:51.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:20:44.677723+0000 '>' 2024-08-19T15:20:44.677723+0000 2024-08-19T15:20:51.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:20:52.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:20:52.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:20:52.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-08-19T15:20:52.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:20:52.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:20:52.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:20:52.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:20:53.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:20:44.677723+0000 '>' 2024-08-19T15:20:44.677723+0000 2024-08-19T15:20:53.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:20:54.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:20:54.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:20:54.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-08-19T15:20:54.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:20:54.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:20:54.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:20:54.095 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:20:54.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:20:52.616805+0000 '>' 2024-08-19T15:20:44.677723+0000 2024-08-19T15:20:54.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:20:54.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3571: TEST_corrupt_scrub_replicated: wait_for_clean 2024-08-19T15:20:54.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:20:54.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:20:54.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:20:54.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:20:54.367 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:20:54.367 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:20:54.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:20:54.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:20:54.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:20:54.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:20:54.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:20:54.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:20:54.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:20:54.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:20:54.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:20:54.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:20:54.726 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:20:54.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:20:54.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:20:54.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:20:54.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=863288426506 2024-08-19T15:20:54.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 863288426506 2024-08-19T15:20:54.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426506' 2024-08-19T15:20:54.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:20:54.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:20:54.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=889058230279 2024-08-19T15:20:54.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 889058230279 2024-08-19T15:20:54.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426506 1-889058230279' 2024-08-19T15:20:54.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:20:54.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-863288426506 2024-08-19T15:20:54.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:20:54.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:20:54.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-863288426506 2024-08-19T15:20:54.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:20:54.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=863288426506 2024-08-19T15:20:54.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 863288426506' 2024-08-19T15:20:54.936 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 863288426506 2024-08-19T15:20:54.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:20:55.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426505 -lt 863288426506 2024-08-19T15:20:55.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:20:56.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:20:56.203 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:20:56.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426506 -lt 863288426506 2024-08-19T15:20:56.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:20:56.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-889058230279 2024-08-19T15:20:56.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:20:56.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:20:56.479 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-889058230279 2024-08-19T15:20:56.479 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:20:56.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=889058230279 2024-08-19T15:20:56.480 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 889058230279 2024-08-19T15:20:56.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 889058230279' 2024-08-19T15:20:56.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:20:56.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 889058230279 -lt 889058230279 2024-08-19T15:20:56.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:20:56.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:20:56.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:20:57.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:20:57.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:20:57.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:20:57.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:20:57.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:20:57.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:20:57.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:20:57.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:20:57.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:20:57.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:20:57.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:20:57.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:20:57.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:20:57.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:20:57.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:20:57.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3574: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2024-08-19T15:20:57.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3575: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2024-08-19T15:20:57.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3577: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-08-19T15:20:57.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3578: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-08-19T15:20:57.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3579: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-08-19T15:20:57.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3580: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-08-19T15:20:57.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3582: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2024-08-19T15:20:57.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3588: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-08-19T15:20:58.387 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'csr_pool' removed 2024-08-19T15:20:58.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T15:20:58.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:20:58.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:20:58.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:20:58.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:20:58.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:20:58.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:20:58.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:20:58.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:20:58.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:20:58.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:20:58.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:20:58.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:20:58.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:20:58.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:20:58.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:20:58.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:20:58.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:20:58.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:20:58.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:20:58.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:20:58.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:20:58.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:20:58.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:20:58.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:20:58.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:20:58.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:20:58.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:20:58.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:20:58.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T15:20:58.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T15:20:58.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T15:20:58.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T15:20:58.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:20:58.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:20:58.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:20:58.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:20:58.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:20:58.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:20:58.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:20:58.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:20:58.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:20:58.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:20:58.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:20:58.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:20:58.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:20:58.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:20:58.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:20:58.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:20:58.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:20:58.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:20:58.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:20:58.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:20:58.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:20:58.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:20:58.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:20:58.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:20:58.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:20:58.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:20:58.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:20:58.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:20:58.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T15:20:58.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T15:20:58.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:20:58.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:20:58.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T15:20:58.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T15:20:58.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T15:20:58.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T15:20:58.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T15:20:58.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2024-08-19T15:20:58.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5931: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2024-08-19T15:20:58.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5932: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2024-08-19T15:20:58.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5933: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2024-08-19T15:20:58.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5935: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-08-19T15:20:58.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T15:20:58.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T15:20:58.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T15:20:58.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T15:20:58.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T15:20:58.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-08-19T15:20:58.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T15:20:58.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:20:58.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:20:58.946 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:20:58.946 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:20:58.946 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:20:58.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:20:58.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-08-19T15:20:59.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T15:20:59.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T15:20:59.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:20:59.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:20:59.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T15:20:59.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T15:20:59.123 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:20:59.123 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:20:59.123 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:20:59.124 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:20:59.124 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:20:59.124 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:20:59.124 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:20:59.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:20:59.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T15:20:59.203 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T15:20:59.203 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:20:59.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:20:59.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T15:20:59.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T15:20:59.204 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:20:59.204 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:20:59.204 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:20:59.204 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:20:59.204 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:20:59.204 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:20:59.204 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:20:59.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:20:59.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T15:20:59.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5936: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2024-08-19T15:20:59.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T15:20:59.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T15:20:59.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T15:20:59.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T15:20:59.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T15:20:59.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T15:21:00.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T15:21:00.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:21:00.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:21:00.090 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:21:00.090 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:21:00.090 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:21:00.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:21:00.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:21:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:21:00.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5937: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2024-08-19T15:21:00.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:21:00.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:21:00.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T15:21:00.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:21:00.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:21:00.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:21:00.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:21:00.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:21:00.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:21:00.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:21:00.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:21:00.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:21:00.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:21:00.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:21:00.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:21:00.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:21:00.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:21:00.122 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:21:00.123 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:21:00.123 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:21:00.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:21:00.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:21:00.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:21:00.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:21:00.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:21:00.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:21:00.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:21:00.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:21:00.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:21:00.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:21:00.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:21:00.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:21:00.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:21:00.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:21:00.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:21:00.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b9e39a83-39bc-452b-89d2-e83576169bd8 2024-08-19T15:21:00.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b9e39a83-39bc-452b-89d2-e83576169bd8' 2024-08-19T15:21:00.125 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 b9e39a83-39bc-452b-89d2-e83576169bd8 2024-08-19T15:21:00.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:21:00.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDcYsNm8BBZCBAAp40EyGmMyzUvW47y5i05eg== 2024-08-19T15:21:00.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDcYsNm8BBZCBAAp40EyGmMyzUvW47y5i05eg=="}' 2024-08-19T15:21:00.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b9e39a83-39bc-452b-89d2-e83576169bd8 -i td/osd-scrub-repair/0/new.json 2024-08-19T15:21:00.338 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:21:00.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T15:21:00.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDcYsNm8BBZCBAAp40EyGmMyzUvW47y5i05eg== --osd-uuid b9e39a83-39bc-452b-89d2-e83576169bd8 2024-08-19T15:21:00.377 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:00.374+0000 7fb8c543d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:21:00.388 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:00.386+0000 7fb8c543d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:21:00.389 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:00.387+0000 7fb8c543d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:21:00.390 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:00.388+0000 7fb8c543d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:21:00.390 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:00.388+0000 7fb8c543d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:21:00.390 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:00.388+0000 7fb8c543d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:21:00.496 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:00.494+0000 7fb8c543d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T15:21:06.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T15:21:06.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:21:06.024 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T15:21:06.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T15:21:06.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:21:06.369 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:21:06.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T15:21:06.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:21:06.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:21:06.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:21:06.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:21:06.396 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:06.395+0000 7f6376101640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:21:06.398 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:06.397+0000 7f6376101640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:21:06.399 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:06.398+0000 7f6376101640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:21:06.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T15:21:06.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:21:06.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:21:06.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:21:06.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:21:06.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:06.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:21:06.637 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:21:06.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:06.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:21:06.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:06.931 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:06.930+0000 7f6376101640 -1 Falling back to public interface 2024-08-19T15:21:07.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:07.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:07.903 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:21:07.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:21:07.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:07.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:21:08.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:09.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:09.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:09.174 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:21:09.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:21:09.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:09.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:21:09.254 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:09.253+0000 7f6376101640 -1 osd.0 0 log_to_monitors true 2024-08-19T15:21:09.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:10.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:10.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:10.453 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:21:10.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:21:10.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:10.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:21:10.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:11.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:11.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:11.732 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:21:11.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:21:11.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:11.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:21:12.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:13.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:13.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:13.075 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:21:13.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:21:13.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:13.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:21:13.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:14.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:14.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:14.353 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:21:14.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:21:14.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:14.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:21:14.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:15.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:15.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:15.633 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:21:15.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:21:15.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:15.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:21:15.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:16.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:16.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:16.912 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:21:16.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:21:16.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:16.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:21:17.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:18.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:18.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:18.187 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:21:18.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:21:18.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:18.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:21:18.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:19.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:19.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:19.464 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:21:19.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:21:19.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:19.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:21:19.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:20.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:20.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:20.740 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:21:20.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:21:20.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:20.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:21:21.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:22.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:22.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:22.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:21:22.443 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:21:22.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:22.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:21:23.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:24.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:24.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:24.170 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:21:24.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:21:24.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:24.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:21:24.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:25.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:25.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:25.947 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:21:25.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:21:25.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:25.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:21:26.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:27.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:27.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:27.391 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:21:27.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:21:27.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:27.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:21:27.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:28.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:28.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:28.837 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:21:28.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:21:28.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:28.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:21:29.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:30.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:30.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:30.670 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:21:30.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:21:30.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:30.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:21:31.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:32.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:32.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:32.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:21:32.038 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:21:32.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:32.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:21:32.331 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3323843465,v1:127.0.0.1:6803/3323843465] [v2:127.0.0.1:6804/3323843465,v1:127.0.0.1:6805/3323843465] exists,up b9e39a83-39bc-452b-89d2-e83576169bd8 2024-08-19T15:21:32.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:21:32.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:21:32.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:21:32.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5938: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2024-08-19T15:21:32.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:21:32.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:21:32.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T15:21:32.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:21:32.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:21:32.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:21:32.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:21:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:21:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:21:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:21:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:21:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:21:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:21:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:21:32.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:21:32.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:21:32.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:21:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:21:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:21:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:21:32.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:21:32.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:21:32.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:21:32.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:21:32.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:21:32.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:21:32.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:21:32.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:21:32.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:21:32.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:21:32.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:21:32.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:21:32.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:21:32.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:21:32.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:21:32.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c4842535-28e8-4bf4-b15b-db53046bfb03 2024-08-19T15:21:32.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c4842535-28e8-4bf4-b15b-db53046bfb03' 2024-08-19T15:21:32.339 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 c4842535-28e8-4bf4-b15b-db53046bfb03 2024-08-19T15:21:32.339 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:21:32.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD8YsNmdxwiFRAAnbw30KeQtLKMT6mlD+ApZQ== 2024-08-19T15:21:32.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD8YsNmdxwiFRAAnbw30KeQtLKMT6mlD+ApZQ=="}' 2024-08-19T15:21:32.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c4842535-28e8-4bf4-b15b-db53046bfb03 -i td/osd-scrub-repair/1/new.json 2024-08-19T15:21:32.692 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:21:32.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T15:21:32.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8YsNmdxwiFRAAnbw30KeQtLKMT6mlD+ApZQ== --osd-uuid c4842535-28e8-4bf4-b15b-db53046bfb03 2024-08-19T15:21:32.734 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:32.732+0000 7fcf21010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:21:32.777 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:32.776+0000 7fcf21010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:21:32.778 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:32.777+0000 7fcf21010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:21:32.779 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:32.777+0000 7fcf21010640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:21:32.779 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:32.777+0000 7fcf21010640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:21:32.779 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:32.777+0000 7fcf21010640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:21:32.852 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:32.851+0000 7fcf21010640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T15:21:38.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T15:21:38.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:21:38.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T15:21:38.534 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T15:21:38.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:21:38.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T15:21:38.877 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:21:38.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:21:38.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:21:38.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:21:38.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:21:38.903 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:38.902+0000 7f5d46082640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:21:38.905 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:38.904+0000 7f5d46082640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:21:38.906 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:38.905+0000 7f5d46082640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:21:39.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T15:21:39.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:21:39.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:21:39.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:21:39.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:21:39.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:39.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:21:39.148 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:21:39.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:39.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:21:39.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:39.447 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:39.446+0000 7f5d46082640 -1 Falling back to public interface 2024-08-19T15:21:40.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:40.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:40.415 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:21:40.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:21:40.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:40.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:21:40.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:41.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:41.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:41.685 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:21:41.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:21:41.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:41.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:21:41.725 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:21:41.724+0000 7f5d46082640 -1 osd.1 0 log_to_monitors true 2024-08-19T15:21:42.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:43.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:43.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:43.002 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:21:43.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:21:43.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:43.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:21:43.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:44.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:44.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:44.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:21:44.279 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:21:44.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:44.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:21:44.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:45.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:45.557 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:21:45.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:45.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:21:45.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:45.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:21:45.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:46.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:46.829 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:21:46.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:46.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:21:46.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:46.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:21:47.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:48.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:48.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:48.105 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:21:48.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:21:48.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:48.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:21:48.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:49.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:49.381 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:21:49.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:49.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:21:49.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:49.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:21:49.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:50.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:50.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:50.655 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:21:50.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:21:50.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:50.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:21:50.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:51.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:51.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:51.930 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:21:51.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:21:51.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:51.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:21:52.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:53.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:53.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:53.209 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:21:53.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:21:53.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:53.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:21:53.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:54.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:54.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:54.521 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:21:54.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:21:54.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:54.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:21:54.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:55.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:55.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:55.917 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:21:55.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:21:55.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:55.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:21:56.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:57.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:57.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:57.294 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:21:57.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:21:57.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:57.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:21:58.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:21:59.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:21:59.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:21:59.022 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:21:59.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:21:59.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:21:59.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:21:59.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:22:00.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:22:00.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:22:00.683 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:22:00.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:22:00.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:22:00.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:22:01.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:22:02.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:22:02.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:22:02.327 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:22:02.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:22:02.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:22:02.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:22:03.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:22:04.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:22:04.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:22:04.163 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:22:04.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:22:04.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:22:04.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:22:04.487 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1365313174,v1:127.0.0.1:6811/1365313174] [v2:127.0.0.1:6812/1365313174,v1:127.0.0.1:6813/1365313174] exists,up c4842535-28e8-4bf4-b15b-db53046bfb03 2024-08-19T15:22:04.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:22:04.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:22:04.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:22:04.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5939: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2024-08-19T15:22:04.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T15:22:04.757 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-08-19T15:22:04.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T15:22:04.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T15:22:05.149 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' created 2024-08-19T15:22:05.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:22:06.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T15:22:07.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5940: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-08-19T15:22:07.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:22:07.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:22:07.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:22:07.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:22:07.987 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:22:07.987 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:22:07.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:22:07.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:22:07.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:22:08.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:22:08.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:22:08.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:22:08.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:22:08.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:22:08.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:22:08.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:22:08.359 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:22:08.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:22:08.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:22:08.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:22:08.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-08-19T15:22:08.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-08-19T15:22:08.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-08-19T15:22:08.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:22:08.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:22:08.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-08-19T15:22:08.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-08-19T15:22:08.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-42949672970' 2024-08-19T15:22:08.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:22:08.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-08-19T15:22:08.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:22:08.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:22:08.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-08-19T15:22:08.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:22:08.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-08-19T15:22:08.568 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836493 2024-08-19T15:22:08.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-08-19T15:22:08.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:22:08.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-08-19T15:22:08.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:22:09.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:22:09.836 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:22:10.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-08-19T15:22:10.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:22:11.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:22:11.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:22:11.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-08-19T15:22:11.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:22:11.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-08-19T15:22:11.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:22:11.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:22:11.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-08-19T15:22:11.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:22:11.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-08-19T15:22:11.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-08-19T15:22:11.387 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672970 2024-08-19T15:22:11.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:22:11.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672970 2024-08-19T15:22:11.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:22:11.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:22:11.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:22:12.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T15:22:12.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:22:12.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:22:12.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:22:12.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:22:12.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:22:12.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:22:12.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:22:12.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:22:12.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:22:12.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:22:12.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:22:12.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T15:22:12.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:22:12.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:22:12.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5942: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2024-08-19T15:22:12.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-08-19T15:22:12.979 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'foo' created 2024-08-19T15:22:12.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:22:13.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5943: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2024-08-19T15:22:13.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-08-19T15:22:14.504 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'csr_pool' created 2024-08-19T15:22:14.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:22:15.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5944: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-08-19T15:22:15.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:22:15.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:22:15.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:22:15.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:22:15.519 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:22:15.519 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:22:15.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:22:15.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:22:15.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:22:15.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:22:15.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:22:15.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:22:15.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:22:15.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:22:15.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:22:15.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:22:15.911 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:22:15.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:22:15.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:22:15.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:22:16.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-08-19T15:22:16.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-08-19T15:22:16.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-08-19T15:22:16.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:22:16.013 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:22:16.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-08-19T15:22:16.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-08-19T15:22:16.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-42949672973' 2024-08-19T15:22:16.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:22:16.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-08-19T15:22:16.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:22:16.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:22:16.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-08-19T15:22:16.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:22:16.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-08-19T15:22:16.119 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836495 2024-08-19T15:22:16.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-08-19T15:22:16.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:22:16.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836495 2024-08-19T15:22:16.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:22:16.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-08-19T15:22:16.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:22:16.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:22:16.396 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-08-19T15:22:16.396 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:22:16.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-08-19T15:22:16.397 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672973 2024-08-19T15:22:16.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-08-19T15:22:16.398 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:22:16.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672973 2024-08-19T15:22:16.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:22:17.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:22:17.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:22:17.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672973 2024-08-19T15:22:17.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:22:18.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:22:18.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:22:19.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-08-19T15:22:19.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:22:19.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:22:19.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:22:19.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:22:19.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:22:19.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:22:19.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:22:19.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:22:19.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:22:19.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:22:19.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:22:19.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:22:19.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:22:19.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:22:19.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:22:20.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:22:20.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:22:20.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:22:20.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-08-19T15:22:20.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-08-19T15:22:20.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-08-19T15:22:20.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-08-19T15:22:20.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:22:20.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:22:20.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-08-19T15:22:20.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:22:20.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:22:20.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:22:20.637 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:22:20.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:22:21.105 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:22:21.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:22:21.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:22:21.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:22:21.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-08-19T15:22:21.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-08-19T15:22:21.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-08-19T15:22:21.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-08-19T15:22:21.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-08-19T15:22:21.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:22:21.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-08-19T15:22:21.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-08-19T15:22:21.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:22:21.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:22:21.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:22:21.976 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:22:21.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:22:22.345 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:22:22.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:22:22.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:22:22.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:22:22.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-08-19T15:22:22.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-08-19T15:22:22.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2024-08-19T15:22:22.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-08-19T15:22:22.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-08-19T15:22:22.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-08-19T15:22:22.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:22:23.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2024-08-19T15:22:23.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2024-08-19T15:22:23.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-08-19T15:22:23.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-08-19T15:22:23.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-08-19T15:22:23.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:22:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: local primary=1 2024-08-19T15:22:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5957: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2024-08-19T15:22:23.634 INFO:tasks.workunit.client.0.smithi102.stdout:created pool csr_pool snap snap1 2024-08-19T15:22:23.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2024-08-19T15:22:23.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-08-19T15:22:23.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-08-19T15:22:23.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-08-19T15:22:23.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2024-08-19T15:22:23.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=1 2024-08-19T15:22:23.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-08-19T15:22:23.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5968: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2024-08-19T15:22:23.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-08-19T15:22:23.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:22:23.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:22:23.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-08-19T15:22:23.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:22:23.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-08-19T15:22:23.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:22:23.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:22:23.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-08-19T15:22:23.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:22:23.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:22:23.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:22:23.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:22:23.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:22:23.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:22:23.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:22:24.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:22:24.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-08-19T15:22:24.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:22:24.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:22:24.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:22:24.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:22:24.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:22:24.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2024-08-19T15:22:26.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-08-19T15:22:26.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:22:26.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:22:26.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:22:26.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:22:26.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:22:26.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:22:26.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:22:26.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:22:26.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:22:26.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:22:26.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:22:26.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:22:26.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:22:26.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:22:26.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:22:26.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:22:26.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:22:26.064 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:22:26.064 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:22:26.065 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:22:26.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:22:26.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:22:26.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:22:26.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:22:26.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:22:26.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:22:26.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:22:26.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:22:26.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:22:26.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:22:26.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:22:26.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:22:26.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:22:26.067 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:22:26.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:22:26.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:22:26.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:22:26.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:22:26.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:22:26.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:22:26.088 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:26.087+0000 7f69bc8db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:22:26.091 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:26.089+0000 7f69bc8db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:22:26.092 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:26.091+0000 7f69bc8db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:22:26.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:22:26.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:22:26.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:22:26.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:22:26.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:22:26.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:22:26.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:22:26.348 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:22:26.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:22:26.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:22:26.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:26.609+0000 7f69bc8db640 -1 Falling back to public interface 2024-08-19T15:22:26.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:22:27.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:22:27.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:22:27.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:22:27.627 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:22:27.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:22:27.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:22:27.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:22:28.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:22:28.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:22:28.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:22:28.899 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:22:28.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:22:28.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:22:28.980 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:28.979+0000 7f69bc8db640 -1 osd.1 30 log_to_monitors true 2024-08-19T15:22:29.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:22:30.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:22:30.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:22:30.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:22:30.179 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:22:30.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:22:30.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:22:30.456 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 34 up_thru 34 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/3650437379,v1:127.0.0.1:6811/3650437379] [v2:127.0.0.1:6812/3650437379,v1:127.0.0.1:6813/3650437379] exists,up c4842535-28e8-4bf4-b15b-db53046bfb03 2024-08-19T15:22:30.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:22:30.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:22:30.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:22:30.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:22:30.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:22:30.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:22:30.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:22:30.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:22:30.457 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:22:30.457 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:22:30.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:22:30.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:22:30.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:22:30.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:22:30.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:22:30.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:22:30.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:22:30.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:22:30.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:22:30.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:22:30.870 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:22:30.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:22:30.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:22:30.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:22:30.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-08-19T15:22:30.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-08-19T15:22:30.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-08-19T15:22:30.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:22:30.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:22:31.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-08-19T15:22:31.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-08-19T15:22:31.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-146028888066' 2024-08-19T15:22:31.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:22:31.101 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-08-19T15:22:31.101 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:22:31.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:22:31.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-08-19T15:22:31.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:22:31.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-08-19T15:22:31.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-08-19T15:22:31.103 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 21474836499 2024-08-19T15:22:31.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:22:31.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-08-19T15:22:31.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:22:32.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:22:32.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:22:32.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836499 2024-08-19T15:22:32.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:22:32.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-146028888066 2024-08-19T15:22:32.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:22:32.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:22:32.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-146028888066 2024-08-19T15:22:32.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:22:32.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-08-19T15:22:32.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 146028888066' 2024-08-19T15:22:32.652 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 146028888066 2024-08-19T15:22:32.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:22:32.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888066 -lt 146028888066 2024-08-19T15:22:32.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:22:32.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:22:32.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:22:33.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:22:33.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:22:33.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:22:33.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:22:33.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:22:33.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:22:33.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:22:33.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:22:33.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:22:33.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:22:33.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:22:33.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:22:33.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:22:33.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:22:33.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:22:33.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-08-19T15:22:33.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-08-19T15:22:33.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2024-08-19T15:22:33.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=0 2024-08-19T15:22:33.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-08-19T15:22:33.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5973: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2024-08-19T15:22:33.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5974: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-08-19T15:22:33.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:22:33.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:22:33.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:22:33.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:22:33.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-08-19T15:22:33.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:22:33.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:22:33.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:22:33.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:22:33.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:22:33.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:22:33.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:22:33.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:22:33.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:22:33.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:22:34.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:22:34.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-08-19T15:22:34.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:22:34.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:22:34.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:22:34.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:22:34.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:22:34.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2024-08-19T15:22:35.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:22:35.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:22:35.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:22:35.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:22:35.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:22:35.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:22:35.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:22:35.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:22:35.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:22:35.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:22:35.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:22:35.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:22:35.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:22:35.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:22:35.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:22:35.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:22:35.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:22:35.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:22:35.978 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:22:35.978 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:22:35.978 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:22:35.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:22:35.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:22:35.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:22:35.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:22:35.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:22:35.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:22:35.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:22:35.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:22:35.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:22:35.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:22:35.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:22:35.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:22:35.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:22:35.979 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:22:35.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:22:35.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:22:35.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:22:35.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:22:35.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:22:35.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:22:36.007 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:36.005+0000 7f5c05b4a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:22:36.009 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:36.008+0000 7f5c05b4a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:22:36.011 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:36.010+0000 7f5c05b4a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:22:36.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:22:36.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:22:36.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:22:36.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:22:36.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:22:36.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:22:36.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:22:36.356 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:22:36.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:22:36.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:22:36.526 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:36.525+0000 7f5c05b4a640 -1 Falling back to public interface 2024-08-19T15:22:36.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:22:37.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:22:37.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:22:37.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:22:37.660 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:22:37.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:22:37.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:22:37.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:22:38.890 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:38.888+0000 7f5c05b4a640 -1 osd.0 36 log_to_monitors true 2024-08-19T15:22:38.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:22:38.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:22:38.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:22:38.935 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:22:38.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:22:38.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:22:39.285 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 40 up_thru 34 down_at 37 last_clean_interval [5,36) [v2:127.0.0.1:6802/2487505257,v1:127.0.0.1:6803/2487505257] [v2:127.0.0.1:6804/2487505257,v1:127.0.0.1:6805/2487505257] exists,up b9e39a83-39bc-452b-89d2-e83576169bd8 2024-08-19T15:22:39.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:22:39.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:22:39.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:22:39.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:22:39.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:22:39.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:22:39.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:22:39.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:22:39.287 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:22:39.287 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:22:39.287 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:22:39.287 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:22:39.287 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:22:39.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:22:39.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:22:39.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:22:39.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:22:39.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:22:39.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:22:39.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:22:39.765 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:22:39.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:22:39.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:22:39.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:22:39.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-08-19T15:22:39.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-08-19T15:22:39.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842' 2024-08-19T15:22:39.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:22:39.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:22:40.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888069 2024-08-19T15:22:40.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888069 2024-08-19T15:22:40.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-146028888069' 2024-08-19T15:22:40.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:22:40.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691842 2024-08-19T15:22:40.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:22:40.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:22:40.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691842 2024-08-19T15:22:40.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:22:40.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-08-19T15:22:40.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2024-08-19T15:22:40.051 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 171798691842 2024-08-19T15:22:40.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:22:40.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-08-19T15:22:40.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:22:40.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-146028888069 2024-08-19T15:22:40.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:22:40.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:22:40.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-146028888069 2024-08-19T15:22:40.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:22:40.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888069 2024-08-19T15:22:40.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 146028888069' 2024-08-19T15:22:40.346 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 146028888069 2024-08-19T15:22:40.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:22:40.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888069 -lt 146028888069 2024-08-19T15:22:40.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:22:40.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:22:40.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:22:40.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:22:40.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:22:40.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:22:40.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:22:40.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:22:40.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:22:40.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:22:40.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:22:41.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:22:41.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:22:41.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:22:41.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:22:41.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:22:41.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:22:41.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:22:41.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5979: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2024-08-19T15:22:41.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5981: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2024-08-19T15:22:41.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-08-19T15:22:41.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-08-19T15:22:41.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:22:41.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:22:41.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:22:41.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:22:41.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-08-19T15:22:14.386702+0000 2024-08-19T15:22:41.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-08-19T15:22:42.077 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 3.0 on osd.1 to scrub 2024-08-19T15:22:42.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-08-19T15:22:14.386702+0000 2024-08-19T15:22:42.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-08-19T15:22:42.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:22:14.386702+0000 2024-08-19T15:22:42.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:22:42.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:22:42.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:22:42.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-08-19T15:22:42.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:22:42.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:22:42.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:22:42.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:22:42.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:22:14.386702+0000 '>' 2024-08-19T15:22:14.386702+0000 2024-08-19T15:22:42.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:22:43.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:22:43.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:22:43.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-08-19T15:22:43.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:22:43.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:22:43.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:22:43.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:22:43.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:22:14.386702+0000 '>' 2024-08-19T15:22:14.386702+0000 2024-08-19T15:22:43.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:22:44.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:22:44.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:22:44.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-08-19T15:22:44.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:22:44.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:22:44.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:22:44.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:22:44.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:22:14.386702+0000 '>' 2024-08-19T15:22:14.386702+0000 2024-08-19T15:22:44.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:22:45.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:22:45.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:22:45.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-08-19T15:22:45.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-08-19T15:22:45.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:22:45.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:22:45.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-08-19T15:22:46.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:22:43.051417+0000 '>' 2024-08-19T15:22:14.386702+0000 2024-08-19T15:22:46.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:22:46.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5983: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2024-08-19T15:22:46.196 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2024-08-19T15:22:46.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2024-08-19T15:22:46.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2024-08-19T15:22:46.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2024-08-19T15:22:46.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5989: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2024-08-19T15:22:46.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-08-19T15:22:46.234 INFO:tasks.workunit.client.0.smithi102.stderr: . as $in 2024-08-19T15:22:46.234 INFO:tasks.workunit.client.0.smithi102.stderr: | if type == "object" then 2024-08-19T15:22:46.234 INFO:tasks.workunit.client.0.smithi102.stderr: reduce keys[] as $key 2024-08-19T15:22:46.234 INFO:tasks.workunit.client.0.smithi102.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-08-19T15:22:46.234 INFO:tasks.workunit.client.0.smithi102.stderr: elif type == "array" then map( walk(f) ) | f 2024-08-19T15:22:46.234 INFO:tasks.workunit.client.0.smithi102.stderr: else f 2024-08-19T15:22:46.234 INFO:tasks.workunit.client.0.smithi102.stderr: end; 2024-08-19T15:22:46.234 INFO:tasks.workunit.client.0.smithi102.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-08-19T15:22:46.235 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-08-19T15:22:46.235 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-08-19T15:22:46.235 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.version) else . end) 2024-08-19T15:22:46.235 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-08-19T15:22:46.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-08-19T15:22:46.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-08-19T15:22:46.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-08-19T15:22:46.256 INFO:tasks.workunit.client.0.smithi102.stderr: . as $in 2024-08-19T15:22:46.257 INFO:tasks.workunit.client.0.smithi102.stderr: | if type == "object" then 2024-08-19T15:22:46.257 INFO:tasks.workunit.client.0.smithi102.stderr: reduce keys[] as $key 2024-08-19T15:22:46.257 INFO:tasks.workunit.client.0.smithi102.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-08-19T15:22:46.257 INFO:tasks.workunit.client.0.smithi102.stderr: elif type == "array" then map( walk(f) ) | f 2024-08-19T15:22:46.257 INFO:tasks.workunit.client.0.smithi102.stderr: else f 2024-08-19T15:22:46.257 INFO:tasks.workunit.client.0.smithi102.stderr: end; 2024-08-19T15:22:46.257 INFO:tasks.workunit.client.0.smithi102.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-08-19T15:22:46.257 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-08-19T15:22:46.257 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-08-19T15:22:46.257 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.version) else . end) 2024-08-19T15:22:46.257 INFO:tasks.workunit.client.0.smithi102.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-08-19T15:22:46.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-08-19T15:22:46.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-08-19T15:22:46.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6158: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-08-19T15:22:46.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-08-19T15:22:46.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6159: TEST_corrupt_snapset_scrub_rep: test no = yes 2024-08-19T15:22:46.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6164: TEST_corrupt_snapset_scrub_rep: test '' = yes 2024-08-19T15:22:46.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6169: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2024-08-19T15:22:46.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6170: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2024-08-19T15:22:46.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6171: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2024-08-19T15:22:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6172: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2024-08-19T15:22:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6173: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2024-08-19T15:22:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6174: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-08-19T15:22:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6175: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2024-08-19T15:22:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6176: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2024-08-19T15:22:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-08-19T15:22:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-08-19T15:22:46.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-08-19T15:22:46.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-08-19T15:22:46.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-08-19T15:22:46.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2024-08-19T15:22:46.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-08-19T15:22:46.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-08-19T15:22:46.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-08-19T15:22:46.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-08-19T15:22:46.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-08-19T15:22:46.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2024-08-19T15:22:46.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6187: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2024-08-19T15:22:46.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6193: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-08-19T15:22:46.653 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'csr_pool' removed 2024-08-19T15:22:46.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T15:22:46.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:22:46.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:22:46.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:22:46.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:22:46.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:22:46.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:22:46.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:22:46.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:22:46.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:22:46.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:22:46.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:22:46.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:22:46.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:22:46.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:22:46.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:22:46.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:22:46.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:22:46.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:22:46.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:22:46.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:22:46.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:22:46.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:22:46.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:22:46.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:22:46.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:22:46.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:22:46.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:22:46.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:22:46.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T15:22:46.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T15:22:46.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T15:22:46.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T15:22:46.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:22:46.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:22:46.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:22:46.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:22:46.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:22:46.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:22:46.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:22:46.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:22:46.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:22:46.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:22:46.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:22:46.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:22:46.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:22:46.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:22:46.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:22:46.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:22:46.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:22:46.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:22:46.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:22:46.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:22:46.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:22:46.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:22:46.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:22:46.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:22:46.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:22:46.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:22:46.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:22:46.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:22:46.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T15:22:46.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T15:22:46.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:22:46.806 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:22:46.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T15:22:46.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T15:22:46.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T15:22:46.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T15:22:46.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T15:22:46.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2024-08-19T15:22:46.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:978: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2024-08-19T15:22:46.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-08-19T15:22:46.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=false 2024-08-19T15:22:46.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-08-19T15:22:46.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-08-19T15:22:46.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T15:22:46.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T15:22:46.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T15:22:46.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T15:22:46.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T15:22:46.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-08-19T15:22:47.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T15:22:47.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:22:47.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:22:47.170 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:22:47.170 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:22:47.170 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:22:47.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:22:47.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T15:22:47.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T15:22:47.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T15:22:47.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:22:47.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:22:47.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T15:22:47.388 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T15:22:47.388 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:22:47.388 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:22:47.388 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:22:47.389 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:22:47.389 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:22:47.389 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:22:47.389 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:22:47.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:22:47.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T15:22:47.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T15:22:47.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:22:47.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:22:47.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T15:22:47.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T15:22:47.468 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:22:47.468 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:22:47.468 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:22:47.469 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:22:47.469 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:22:47.469 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:22:47.469 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:22:47.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:22:47.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T15:22:47.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-08-19T15:22:47.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T15:22:47.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T15:22:47.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T15:22:47.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T15:22:47.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T15:22:47.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T15:22:48.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T15:22:48.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:22:48.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:22:48.355 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:22:48.356 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:22:48.356 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:22:48.356 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:22:48.356 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:22:48.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:22:48.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-08-19T15:22:48.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-08-19T15:22:48.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-08-19T15:22:48.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:22:48.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:22:48.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T15:22:48.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:22:48.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:22:48.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:22:48.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:22:48.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:22:48.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:22:48.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:22:48.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:22:48.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:22:48.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:22:48.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:22:48.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:22:48.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:22:48.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:22:48.391 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:22:48.391 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:22:48.391 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:22:48.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:22:48.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:22:48.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:22:48.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:22:48.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:22:48.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:22:48.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:22:48.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:22:48.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:22:48.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:22:48.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:22:48.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:22:48.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:22:48.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:22:48.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:22:48.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aef395e2-a8af-46dd-a8de-671a7857f546 2024-08-19T15:22:48.395 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 aef395e2-a8af-46dd-a8de-671a7857f546 2024-08-19T15:22:48.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 aef395e2-a8af-46dd-a8de-671a7857f546' 2024-08-19T15:22:48.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:22:48.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBIY8NmJrA5GBAAMe6cUs3Vys/Nn6GehoDcIg== 2024-08-19T15:22:48.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBIY8NmJrA5GBAAMe6cUs3Vys/Nn6GehoDcIg=="}' 2024-08-19T15:22:48.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aef395e2-a8af-46dd-a8de-671a7857f546 -i td/osd-scrub-repair/0/new.json 2024-08-19T15:22:48.611 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:22:48.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T15:22:48.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBIY8NmJrA5GBAAMe6cUs3Vys/Nn6GehoDcIg== --osd-uuid aef395e2-a8af-46dd-a8de-671a7857f546 2024-08-19T15:22:48.651 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:48.650+0000 7f3d8e21d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:22:48.662 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:48.661+0000 7f3d8e21d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:22:48.663 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:48.662+0000 7f3d8e21d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:22:48.663 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:48.662+0000 7f3d8e21d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:22:48.664 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:48.663+0000 7f3d8e21d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:22:48.664 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:48.663+0000 7f3d8e21d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:22:48.754 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:48.753+0000 7f3d8e21d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T15:22:54.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T15:22:54.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:22:54.261 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T15:22:54.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T15:22:54.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:22:54.608 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:22:54.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T15:22:54.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:22:54.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:22:54.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:22:54.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:22:54.635 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:54.634+0000 7f735ae1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:22:54.637 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:54.636+0000 7f735ae1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:22:54.638 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:54.637+0000 7f735ae1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:22:54.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T15:22:54.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:22:54.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:22:54.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:22:54.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:22:54.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:22:54.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:22:54.875 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:22:54.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:22:54.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:22:55.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:22:55.170 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:55.169+0000 7f735ae1f640 -1 Falling back to public interface 2024-08-19T15:22:56.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:22:56.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:22:56.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:22:56.141 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:22:56.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:22:56.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:22:56.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:22:57.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:22:57.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:22:57.412 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:22:57.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:22:57.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:22:57.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:22:57.494 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:22:57.493+0000 7f735ae1f640 -1 osd.0 0 log_to_monitors true 2024-08-19T15:22:57.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:22:58.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:22:58.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:22:58.690 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:22:58.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:22:58.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:22:58.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:22:58.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:22:59.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:22:59.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:22:59.964 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:22:59.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:22:59.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:22:59.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:23:00.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:01.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:01.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:01.245 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:23:01.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:23:01.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:01.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:23:01.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:02.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:02.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:02.520 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:23:02.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:23:02.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:02.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:23:02.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:03.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:03.794 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:23:03.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:03.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:23:03.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:03.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:23:04.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:05.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:05.071 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:23:05.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:05.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:23:05.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:05.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:23:05.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:06.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:06.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:06.433 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:23:06.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:23:06.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:06.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:23:06.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:07.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:07.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:07.707 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:23:07.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:23:07.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:07.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:23:08.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:09.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:09.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:09.029 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:23:09.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:23:09.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:09.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:23:09.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:10.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:10.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:10.644 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:23:10.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:23:10.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:10.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:23:11.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:12.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:12.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:12.327 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:23:12.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:23:12.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:12.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:23:13.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:14.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:14.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:14.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:23:14.023 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:23:14.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:14.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:23:14.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:15.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:15.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:15.449 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:23:15.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:23:15.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:15.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:23:15.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:16.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:16.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:16.879 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:23:16.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:23:16.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:16.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:23:17.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:18.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:18.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:18.404 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:23:18.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:23:18.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:18.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:23:18.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:19.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:19.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:19.915 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:23:19.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:23:19.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:19.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:23:20.184 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4065472344,v1:127.0.0.1:6803/4065472344] [v2:127.0.0.1:6804/4065472344,v1:127.0.0.1:6805/4065472344] exists,up aef395e2-a8af-46dd-a8de-671a7857f546 2024-08-19T15:23:20.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:23:20.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:23:20.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:23:20.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-08-19T15:23:20.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-08-19T15:23:20.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:23:20.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:23:20.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T15:23:20.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:23:20.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:23:20.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:23:20.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:23:20.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:23:20.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:23:20.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:23:20.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:23:20.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:23:20.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:23:20.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:23:20.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:23:20.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:23:20.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:23:20.190 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:23:20.190 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:23:20.190 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:23:20.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:23:20.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:23:20.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:23:20.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:23:20.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:23:20.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:23:20.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:23:20.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:23:20.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:23:20.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:23:20.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:23:20.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:23:20.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:23:20.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:23:20.192 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:23:20.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=62b7dab7-1fd2-41b9-945a-c4f8f5fe31ff 2024-08-19T15:23:20.192 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 62b7dab7-1fd2-41b9-945a-c4f8f5fe31ff 2024-08-19T15:23:20.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 62b7dab7-1fd2-41b9-945a-c4f8f5fe31ff' 2024-08-19T15:23:20.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:23:20.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBoY8NmPLU6DBAAuGcQ/Olvo3x6oxctOkM4ow== 2024-08-19T15:23:20.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBoY8NmPLU6DBAAuGcQ/Olvo3x6oxctOkM4ow=="}' 2024-08-19T15:23:20.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 62b7dab7-1fd2-41b9-945a-c4f8f5fe31ff -i td/osd-scrub-repair/1/new.json 2024-08-19T15:23:20.666 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:23:20.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T15:23:20.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBoY8NmPLU6DBAAuGcQ/Olvo3x6oxctOkM4ow== --osd-uuid 62b7dab7-1fd2-41b9-945a-c4f8f5fe31ff 2024-08-19T15:23:20.707 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:20.706+0000 7f7b3eb4e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:23:20.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:20.741+0000 7f7b3eb4e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:23:20.743 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:20.742+0000 7f7b3eb4e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:23:20.744 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:20.743+0000 7f7b3eb4e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:23:20.744 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:20.743+0000 7f7b3eb4e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:23:20.744 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:20.743+0000 7f7b3eb4e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:23:20.817 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:20.816+0000 7f7b3eb4e640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T15:23:26.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T15:23:26.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:23:26.502 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T15:23:26.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T15:23:26.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:23:26.851 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:23:26.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T15:23:26.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:23:26.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:23:26.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:23:26.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:23:26.878 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:26.876+0000 7f1738b40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:23:26.880 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:26.878+0000 7f1738b40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:23:26.881 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:26.879+0000 7f1738b40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:23:27.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T15:23:27.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:23:27.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:23:27.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:23:27.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:23:27.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:27.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:23:27.118 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:23:27.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:27.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:23:27.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:27.407 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:27.405+0000 7f1738b40640 -1 Falling back to public interface 2024-08-19T15:23:28.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:28.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:28.383 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:23:28.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:23:28.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:28.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:23:28.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:29.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:29.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:29.678 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:23:29.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:23:29.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:29.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:23:29.757 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:29.755+0000 7f1738b40640 -1 osd.1 0 log_to_monitors true 2024-08-19T15:23:29.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:30.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:30.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:30.958 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:23:30.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:23:30.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:30.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:23:31.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:32.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:32.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:32.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:23:32.257 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:23:32.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:32.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:23:32.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:33.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:33.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:33.555 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:23:33.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:23:33.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:33.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:23:33.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:34.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:34.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:34.832 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:23:34.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:23:34.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:34.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:23:35.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:36.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:36.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:36.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:23:36.111 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:23:36.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:36.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:23:36.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:37.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:37.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:37.389 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:23:37.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:23:37.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:37.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:23:37.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:38.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:38.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:38.701 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:23:38.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:23:38.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:38.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:23:38.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:39.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:39.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:39.978 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:23:39.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:23:39.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:39.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:23:40.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:41.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:41.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:41.260 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:23:41.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:23:41.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:41.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:23:42.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:43.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:43.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:43.010 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:23:43.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:23:43.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:43.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:23:43.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:44.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:44.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:44.747 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:23:44.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:23:44.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:44.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:23:45.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:46.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:46.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:46.766 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:23:46.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:23:46.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:46.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:23:47.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:48.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:48.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:48.460 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:23:48.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:23:48.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:48.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:23:48.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:49.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:49.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:49.851 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:23:49.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:23:49.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:49.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:23:50.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:51.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:51.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:51.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:23:51.426 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:23:51.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:51.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:23:51.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:52.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:23:52.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:52.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:23:52.720 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:23:52.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:52.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:23:52.993 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1004426796,v1:127.0.0.1:6811/1004426796] [v2:127.0.0.1:6812/1004426796,v1:127.0.0.1:6813/1004426796] exists,up 62b7dab7-1fd2-41b9-945a-c4f8f5fe31ff 2024-08-19T15:23:52.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:23:52.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:23:52.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:23:52.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-08-19T15:23:52.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-08-19T15:23:52.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:23:52.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:23:52.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T15:23:52.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:23:52.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:23:52.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:23:52.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:23:52.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:23:52.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:23:52.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T15:23:52.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T15:23:52.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:23:52.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:23:52.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:23:52.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:23:52.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:23:52.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:23:52.999 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:23:52.999 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:23:52.999 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:23:52.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:23:53.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:23:53.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:23:53.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:23:53.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:23:53.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:23:53.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:23:53.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:23:53.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:23:53.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:23:53.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:23:53.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:23:53.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:23:53.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T15:23:53.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:23:53.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d94f7376-519d-450e-b70c-0cf2e6cd8644 2024-08-19T15:23:53.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d94f7376-519d-450e-b70c-0cf2e6cd8644' 2024-08-19T15:23:53.001 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 d94f7376-519d-450e-b70c-0cf2e6cd8644 2024-08-19T15:23:53.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:23:53.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCJY8NmTo30ABAAuEzzILA2awHl/hEWnnQIHg== 2024-08-19T15:23:53.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCJY8NmTo30ABAAuEzzILA2awHl/hEWnnQIHg=="}' 2024-08-19T15:23:53.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d94f7376-519d-450e-b70c-0cf2e6cd8644 -i td/osd-scrub-repair/2/new.json 2024-08-19T15:23:53.344 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:23:53.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-08-19T15:23:53.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCJY8NmTo30ABAAuEzzILA2awHl/hEWnnQIHg== --osd-uuid d94f7376-519d-450e-b70c-0cf2e6cd8644 2024-08-19T15:23:53.384 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:53.383+0000 7fe3c8a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:23:53.403 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:53.402+0000 7fe3c8a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:23:53.404 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:53.403+0000 7fe3c8a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:23:53.405 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:53.404+0000 7fe3c8a0c640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:23:53.405 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:53.404+0000 7fe3c8a0c640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:23:53.405 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:53.404+0000 7fe3c8a0c640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:23:53.470 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:53.469+0000 7fe3c8a0c640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-08-19T15:23:58.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-08-19T15:23:58.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:23:58.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T15:23:58.952 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T15:23:58.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:23:59.298 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T15:23:59.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T15:23:59.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:23:59.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:23:59.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:23:59.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:23:59.324 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:59.323+0000 7f572b424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:23:59.326 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:59.325+0000 7f572b424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:23:59.327 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:59.326+0000 7f572b424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:23:59.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T15:23:59.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:23:59.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T15:23:59.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:23:59.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:23:59.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:23:59.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:23:59.565 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:23:59.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:23:59.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:23:59.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:23:59.864 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:23:59.863+0000 7f572b424640 -1 Falling back to public interface 2024-08-19T15:24:00.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:00.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:00.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:24:00.831 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:24:00.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:00.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:24:01.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:02.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:02.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:02.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:24:02.099 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:24:02.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:02.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:24:02.219 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:24:02.218+0000 7f572b424640 -1 osd.2 0 log_to_monitors true 2024-08-19T15:24:02.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:03.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:03.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:03.420 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:24:03.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:24:03.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:03.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:24:03.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:04.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:04.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:04.699 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:24:04.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:24:04.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:04.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:24:05.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:06.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:06.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:06.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:24:06.067 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:24:06.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:06.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:24:06.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:07.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:07.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:07.405 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:24:07.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:24:07.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:07.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:24:07.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:08.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:08.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:08.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:24:08.680 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:24:08.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:08.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:24:09.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:10.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:10.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:10.136 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:24:10.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:24:10.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:10.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:24:10.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:11.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:11.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:11.410 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:24:11.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:24:11.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:11.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:24:11.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:12.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:12.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:12.686 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:24:12.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:24:12.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:12.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:24:13.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:14.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:14.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:14.056 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:24:14.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:24:14.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:14.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:24:14.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:15.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:15.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:15.586 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:24:15.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:24:15.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:15.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:24:16.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:17.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:17.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:17.183 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:24:17.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:24:17.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:17.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:24:17.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:18.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:18.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:18.744 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:24:18.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:24:18.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:18.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:24:19.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:20.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:20.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:20.127 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:24:20.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:24:20.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:20.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:24:20.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:21.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:21.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:21.512 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:24:21.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:24:21.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:21.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:24:22.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:23.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:23.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:23.029 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:24:23.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:24:23.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:23.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:24:23.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:24.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:24.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:24.455 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:24:24.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:24:24.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:24.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:24:24.746 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4189899787,v1:127.0.0.1:6819/4189899787] [v2:127.0.0.1:6820/4189899787,v1:127.0.0.1:6821/4189899787] exists,up d94f7376-519d-450e-b70c-0cf2e6cd8644 2024-08-19T15:24:24.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:24:24.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:24:24.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:24:24.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-08-19T15:24:24.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T15:24:25.003 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-08-19T15:24:25.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T15:24:25.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T15:24:25.455 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' created 2024-08-19T15:24:25.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:24:26.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T15:24:28.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-08-19T15:24:28.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:24:28.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:24:28.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:24:28.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:24:28.546 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:24:28.547 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:24:28.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:24:28.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:24:28.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:24:28.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:24:28.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:24:28.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:24:28.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:24:28.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:24:28.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:24:28.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:24:28.960 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:24:28.960 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:24:28.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:24:28.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:24:28.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:24:29.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-08-19T15:24:29.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-08-19T15:24:29.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-08-19T15:24:29.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:24:29.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:24:29.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-08-19T15:24:29.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-08-19T15:24:29.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672971' 2024-08-19T15:24:29.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:24:29.166 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:24:29.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-08-19T15:24:29.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-08-19T15:24:29.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672971 2-64424509444' 2024-08-19T15:24:29.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:24:29.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-08-19T15:24:29.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:24:29.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:24:29.269 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-08-19T15:24:29.269 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:24:29.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-08-19T15:24:29.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-08-19T15:24:29.271 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836500 2024-08-19T15:24:29.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:24:29.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-08-19T15:24:29.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:24:30.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:24:30.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:24:30.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836500 2024-08-19T15:24:30.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:24:30.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-08-19T15:24:30.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:24:30.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:24:30.820 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-08-19T15:24:30.820 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:24:30.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-08-19T15:24:30.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-08-19T15:24:30.822 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672971 2024-08-19T15:24:30.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:24:31.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672971 2024-08-19T15:24:31.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:24:31.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-08-19T15:24:31.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:24:31.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:24:31.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-08-19T15:24:31.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:24:31.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-08-19T15:24:31.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-08-19T15:24:31.093 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509444 2024-08-19T15:24:31.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:24:31.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509444 2024-08-19T15:24:31.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:24:31.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:24:31.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:24:31.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T15:24:31.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:24:31.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:24:31.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:24:31.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:24:31.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:24:31.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:24:31.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:24:31.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:24:31.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:24:31.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:24:31.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:24:32.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T15:24:32.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:24:32.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:24:32.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-08-19T15:24:32.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-08-19T15:24:32.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-08-19T15:24:32.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-08-19T15:24:32.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-08-19T15:24:32.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-08-19T15:24:32.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-08-19T15:24:32.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-08-19T15:24:33.412 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'ecpool' created 2024-08-19T15:24:33.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:24:34.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-08-19T15:24:34.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-08-19T15:24:34.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:24:34.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:24:34.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:24:34.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:24:34.427 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:24:34.427 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:24:34.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:24:34.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:24:34.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:24:34.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:24:34.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:24:34.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:24:34.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:24:34.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:24:34.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:24:34.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:24:34.882 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:24:34.882 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:24:34.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:24:34.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:24:34.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:24:34.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-08-19T15:24:34.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-08-19T15:24:34.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-08-19T15:24:34.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:24:34.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:24:35.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-08-19T15:24:35.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-08-19T15:24:35.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672973' 2024-08-19T15:24:35.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:24:35.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:24:35.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-08-19T15:24:35.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-08-19T15:24:35.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672973 2-64424509447' 2024-08-19T15:24:35.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:24:35.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-08-19T15:24:35.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:24:35.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:24:35.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-08-19T15:24:35.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:24:35.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-08-19T15:24:35.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-08-19T15:24:35.196 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836503 2024-08-19T15:24:35.196 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:24:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836503 2024-08-19T15:24:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:24:36.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:24:36.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:24:36.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-08-19T15:24:36.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:24:36.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-08-19T15:24:36.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:24:36.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:24:36.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-08-19T15:24:36.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:24:36.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-08-19T15:24:36.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-08-19T15:24:36.760 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672973 2024-08-19T15:24:36.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:24:37.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-08-19T15:24:37.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:24:37.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-08-19T15:24:37.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:24:37.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:24:37.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-08-19T15:24:37.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:24:37.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-08-19T15:24:37.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-08-19T15:24:37.031 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509447 2024-08-19T15:24:37.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:24:37.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-08-19T15:24:37.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:24:37.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:24:37.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:24:37.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:24:37.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:24:37.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:24:37.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:24:37.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:24:37.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:24:37.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:24:37.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:24:37.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:24:37.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:24:37.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:24:37.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:24:38.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:24:38.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:24:38.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:24:38.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-08-19T15:24:38.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-08-19T15:24:38.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:24:38.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:24:38.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-08-19T15:24:38.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:24:38.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:24:38.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:24:38.736 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:24:38.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:24:39.125 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:24:39.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:24:39.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:24:39.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:24:39.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-08-19T15:24:39.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-08-19T15:24:39.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-08-19T15:24:39.713 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-08-19T15:24:39.713 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-19T15:24:40.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-08-19T15:24:40.028 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:24:40.028 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-08-19T15:24:40.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-08-19T15:24:40.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-08-19T15:24:40.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-08-19T15:24:40.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-08-19T15:24:40.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:24:40.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:24:40.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-08-19T15:24:40.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:24:40.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:24:40.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:24:40.422 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:24:40.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:24:40.804 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:24:40.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:24:40.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:24:40.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:24:41.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-08-19T15:24:41.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-08-19T15:24:41.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-08-19T15:24:41.093 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-08-19T15:24:41.093 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-19T15:24:41.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-08-19T15:24:41.365 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:24:41.365 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-08-19T15:24:41.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-08-19T15:24:41.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-08-19T15:24:41.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-08-19T15:24:41.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-08-19T15:24:41.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-08-19T15:24:41.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:24:41.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:24:41.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:24:41.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:24:41.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:24:41.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:24:41.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:24:41.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-08-19T15:24:41.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:24:41.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:24:41.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:24:41.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:24:41.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:24:41.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:24:42.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:24:42.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-08-19T15:24:42.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-08-19T15:24:42.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:24:42.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:24:42.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:24:42.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:24:42.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:24:43.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:24:43.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-08-19T15:24:43.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-08-19T15:24:44.356 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-08-19T15:24:44.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-08-19T15:24:44.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-08-19T15:24:45.960 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-08-19T15:24:46.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-08-19T15:24:46.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-08-19T15:24:47.222 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-08-19T15:24:47.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-08-19T15:24:47.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-08-19T15:24:48.914 INFO:tasks.workunit.client.0.smithi102.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-08-19T15:24:49.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-08-19T15:24:49.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-08-19T15:24:49.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-08-19T15:24:49.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:24:49.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:24:49.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:24:49.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:24:49.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:24:49.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:24:49.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:24:49.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:24:49.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:24:49.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:24:49.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:24:49.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:24:49.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:24:49.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:24:49.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:24:49.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:24:49.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:24:49.438 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:24:49.438 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:24:49.438 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:24:49.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:24:49.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:24:49.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:24:49.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:24:49.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:24:49.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:24:49.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:24:49.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:24:49.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:24:49.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:24:49.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:24:49.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:24:49.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:24:49.440 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:24:49.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:24:49.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:24:49.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:24:49.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:24:49.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:24:49.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:24:49.466 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:24:49.465+0000 7fc6ca843640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:24:49.469 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:24:49.468+0000 7fc6ca843640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:24:49.470 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:24:49.469+0000 7fc6ca843640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:24:49.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:24:49.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:24:49.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:24:49.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:24:49.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:24:49.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:49.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:24:49.719 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:24:49.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:49.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:24:49.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:49.998 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:24:49.998+0000 7fc6ca843640 -1 Falling back to public interface 2024-08-19T15:24:50.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:50.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:50.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:24:50.986 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:24:50.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:50.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:24:51.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:52.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:52.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:52.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:24:52.263 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:24:52.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:52.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:24:52.354 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:24:52.353+0000 7fc6ca843640 -1 osd.0 34 log_to_monitors true 2024-08-19T15:24:52.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:53.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:53.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:53.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:24:53.565 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:24:53.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:53.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:24:53.838 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 40 up_thru 40 down_at 35 last_clean_interval [5,34) [v2:127.0.0.1:6802/3886251632,v1:127.0.0.1:6803/3886251632] [v2:127.0.0.1:6804/3886251632,v1:127.0.0.1:6805/3886251632] exists,up aef395e2-a8af-46dd-a8de-671a7857f546 2024-08-19T15:24:53.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:24:53.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:24:53.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:24:53.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-08-19T15:24:53.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-08-19T15:24:53.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:24:53.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:24:53.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:24:53.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:24:53.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:24:53.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:24:53.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:24:53.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:24:53.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:24:53.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:24:53.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:24:53.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:24:53.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:24:53.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:24:53.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:24:53.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:24:53.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:24:53.840 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:24:53.841 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:24:53.841 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:24:53.841 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:24:53.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:24:53.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:24:53.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:24:53.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:24:53.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:24:53.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:24:53.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:24:53.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:24:53.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:24:53.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:24:53.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:24:53.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:24:53.842 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:24:53.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:24:53.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:24:53.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:24:53.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:24:53.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:24:53.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:24:53.865 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:24:53.864+0000 7f78d6011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:24:53.866 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:24:53.865+0000 7f78d6011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:24:53.868 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:24:53.867+0000 7f78d6011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:24:54.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:24:54.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:24:54.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:24:54.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:24:54.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:24:54.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:54.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:24:54.140 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:24:54.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:54.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:24:54.390 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:24:54.389+0000 7f78d6011640 -1 Falling back to public interface 2024-08-19T15:24:54.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:55.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:55.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:55.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:24:55.449 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:24:55.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:55.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:24:55.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:56.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:56.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:56.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:24:56.726 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:24:56.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:56.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:24:56.783 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:24:56.782+0000 7f78d6011640 -1 osd.1 35 log_to_monitors true 2024-08-19T15:24:57.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:58.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:58.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:58.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:24:58.038 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:24:58.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:58.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:24:58.316 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/3687520232,v1:127.0.0.1:6811/3687520232] [v2:127.0.0.1:6812/3687520232,v1:127.0.0.1:6813/3687520232] exists,up 62b7dab7-1fd2-41b9-945a-c4f8f5fe31ff 2024-08-19T15:24:58.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:24:58.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:24:58.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:24:58.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-08-19T15:24:58.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-08-19T15:24:58.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:24:58.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:24:58.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-19T15:24:58.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:24:58.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:24:58.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:24:58.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:24:58.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:24:58.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:24:58.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T15:24:58.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T15:24:58.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:24:58.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:24:58.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:24:58.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:24:58.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:24:58.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:24:58.318 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:24:58.318 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:24:58.318 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:24:58.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:24:58.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:24:58.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:24:58.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:24:58.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:24:58.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:24:58.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:24:58.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:24:58.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:24:58.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:24:58.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:24:58.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T15:24:58.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-19T15:24:58.319 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.2 2024-08-19T15:24:58.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:24:58.319 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-08-19T15:24:58.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-19T15:24:58.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:24:58.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:24:58.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:24:58.343 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:24:58.342+0000 7f731ba18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:24:58.357 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:24:58.356+0000 7f731ba18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:24:58.359 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:24:58.358+0000 7f731ba18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:24:58.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-19T15:24:58.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:24:58.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T15:24:58.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:24:58.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:24:58.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:58.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:24:58.605 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:24:58.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:58.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:24:58.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:24:58.881 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:24:58.880+0000 7f731ba18640 -1 Falling back to public interface 2024-08-19T15:24:59.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:24:59.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:24:59.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:24:59.874 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:24:59.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:24:59.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:25:00.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:01.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:01.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:01.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:25:01.145 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:25:01.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:01.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:25:01.273 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:01.272+0000 7f731ba18640 -1 osd.2 37 log_to_monitors true 2024-08-19T15:25:01.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:02.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:02.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:02.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:25:02.453 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:25:02.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:02.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:25:02.797 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 up in weight 1 up_from 47 up_thru 47 down_at 38 last_clean_interval [15,37) [v2:127.0.0.1:6818/2228955320,v1:127.0.0.1:6819/2228955320] [v2:127.0.0.1:6820/2228955320,v1:127.0.0.1:6821/2228955320] exists,up d94f7376-519d-450e-b70c-0cf2e6cd8644 2024-08-19T15:25:02.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:25:02.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:25:02.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:25:02.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-08-19T15:25:02.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T15:25:03.149 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' removed 2024-08-19T15:25:03.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T15:25:03.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T15:25:03.619 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' created 2024-08-19T15:25:03.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:25:04.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T15:25:06.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-08-19T15:25:06.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:25:06.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:25:06.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:25:06.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:25:06.155 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:25:06.156 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:25:06.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:25:06.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:25:06.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:25:06.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:25:06.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:25:06.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:25:06.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:25:06.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:25:06.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:25:06.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:25:06.542 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:25:06.542 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:25:06.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:25:06.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:25:06.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:25:06.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691844 2024-08-19T15:25:06.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691844 2024-08-19T15:25:06.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691844' 2024-08-19T15:25:06.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:25:06.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:25:06.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593731 2024-08-19T15:25:06.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593731 2024-08-19T15:25:06.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691844 1-184683593731' 2024-08-19T15:25:06.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:25:06.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:25:06.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462915 2024-08-19T15:25:06.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462915 2024-08-19T15:25:06.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691844 1-184683593731 2-201863462915' 2024-08-19T15:25:06.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:25:06.858 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691844 2024-08-19T15:25:06.858 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:25:06.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:25:06.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691844 2024-08-19T15:25:06.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:25:06.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691844 2024-08-19T15:25:06.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691844' 2024-08-19T15:25:06.861 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 171798691844 2024-08-19T15:25:06.861 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:25:07.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691844 2024-08-19T15:25:07.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:25:08.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:25:08.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:25:08.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691844 2024-08-19T15:25:08.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:25:08.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-184683593731 2024-08-19T15:25:08.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:25:08.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:25:08.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-184683593731 2024-08-19T15:25:08.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:25:08.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593731 2024-08-19T15:25:08.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 184683593731' 2024-08-19T15:25:08.408 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 184683593731 2024-08-19T15:25:08.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:25:08.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593732 -lt 184683593731 2024-08-19T15:25:08.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:25:08.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-201863462915 2024-08-19T15:25:08.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:25:08.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:25:08.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-201863462915 2024-08-19T15:25:08.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:25:08.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462915 2024-08-19T15:25:08.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-08-19T15:25:08.716 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 201863462915 2024-08-19T15:25:08.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:25:08.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462915 -lt 201863462915 2024-08-19T15:25:08.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:25:08.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:25:08.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:25:09.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:25:09.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:25:09.322 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:25:09.322 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:25:09.322 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:25:09.322 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:25:09.322 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:25:09.322 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:25:09.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:25:09.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:25:09.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:25:09.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:25:09.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:25:09.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:25:09.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:25:09.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-08-19T15:25:09.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T15:25:09.959 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-08-19T15:25:09.959 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-08-19T15:25:09.959 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:25:10.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-08-19T15:25:10.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-08-19T15:25:10.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-08-19T15:25:10.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-08-19T15:25:10.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:25:10.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:25:10.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:25:10.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:25:10.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T15:24:33.285589+0000 2024-08-19T15:25:10.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-08-19T15:25:10.839 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-08-19T15:25:10.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-08-19T15:24:33.285589+0000 2024-08-19T15:25:10.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T15:25:10.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:24:33.285589+0000 2024-08-19T15:25:10.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:25:10.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:25:10.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:25:10.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:25:10.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:25:10.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:25:10.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:25:10.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:25:11.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:24:33.285589+0000 '>' 2024-08-19T15:24:33.285589+0000 2024-08-19T15:25:11.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:25:12.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:25:12.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:25:12.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:25:12.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:25:12.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:25:12.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:25:12.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:25:12.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:25:11.209746+0000 '>' 2024-08-19T15:24:33.285589+0000 2024-08-19T15:25:12.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:25:12.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-08-19T15:25:12.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-08-19T15:25:12.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-08-19T15:25:12.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-08-19T15:25:12.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-08-19T15:25:12.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-08-19T15:25:12.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-08-19T15:25:12.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-08-19T15:25:12.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-08-19T15:25:12.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T15:25:12.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:25:12.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:25:12.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:25:12.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:25:12.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:25:12.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:25:12.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:25:12.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:25:12.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:25:12.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:25:12.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:25:12.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:25:12.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:25:12.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:25:12.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:25:12.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:25:12.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:25:12.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:25:12.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:25:12.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:25:12.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:25:12.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:25:12.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:25:12.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:25:12.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:25:12.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:25:12.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:25:12.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:25:12.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T15:25:12.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T15:25:12.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T15:25:12.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T15:25:12.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:25:12.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:25:12.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:25:12.623 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:25:12.623 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:25:12.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:25:12.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:25:12.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:25:12.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:25:12.624 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:25:12.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:25:12.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:25:12.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:25:12.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:25:12.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:25:12.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:25:12.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:25:12.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:25:12.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:25:12.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:25:12.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:25:12.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:25:12.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:25:12.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:25:12.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:25:12.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:25:12.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:25:12.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:25:12.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T15:25:12.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T15:25:12.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:25:12.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:25:12.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T15:25:12.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T15:25:12.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T15:25:12.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T15:25:12.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T15:25:12.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2024-08-19T15:25:12.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2024-08-19T15:25:12.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:983: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2024-08-19T15:25:12.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-08-19T15:25:12.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=true 2024-08-19T15:25:12.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-08-19T15:25:12.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-08-19T15:25:12.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T15:25:12.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T15:25:12.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T15:25:12.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T15:25:12.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T15:25:12.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-08-19T15:25:13.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T15:25:13.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:25:13.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:25:13.022 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:25:13.022 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:25:13.022 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:25:13.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:25:13.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T15:25:13.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T15:25:13.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T15:25:13.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:25:13.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:25:13.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T15:25:13.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T15:25:13.199 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:25:13.199 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:25:13.199 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:25:13.199 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:25:13.199 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:25:13.199 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:25:13.199 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:25:13.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:25:13.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T15:25:13.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T15:25:13.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:25:13.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:25:13.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T15:25:13.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T15:25:13.279 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:25:13.279 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:25:13.279 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:25:13.279 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:25:13.280 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:25:13.280 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:25:13.280 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:25:13.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:25:13.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T15:25:13.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-08-19T15:25:13.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T15:25:13.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T15:25:13.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T15:25:13.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T15:25:13.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T15:25:13.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T15:25:14.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T15:25:14.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:25:14.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:25:14.176 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:25:14.176 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:25:14.176 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:25:14.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:25:14.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:25:14.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:25:14.208 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-08-19T15:25:14.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-08-19T15:25:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-08-19T15:25:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:25:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:25:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T15:25:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:25:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:25:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:25:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:25:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:25:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:25:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:25:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:25:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:25:14.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:25:14.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:25:14.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:25:14.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:25:14.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:25:14.210 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:25:14.210 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:25:14.210 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:25:14.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:25:14.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:25:14.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:25:14.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:25:14.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:25:14.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:25:14.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:25:14.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:25:14.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:25:14.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:25:14.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:25:14.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:25:14.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:25:14.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:25:14.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:25:14.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1e09b8f4-7959-48a9-905d-843749c4027b 2024-08-19T15:25:14.216 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 1e09b8f4-7959-48a9-905d-843749c4027b 2024-08-19T15:25:14.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1e09b8f4-7959-48a9-905d-843749c4027b' 2024-08-19T15:25:14.217 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:25:14.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDaY8Nm6YGBDRAAPaZWSE3OMPmWa5cP+mrmCA== 2024-08-19T15:25:14.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDaY8Nm6YGBDRAAPaZWSE3OMPmWa5cP+mrmCA=="}' 2024-08-19T15:25:14.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1e09b8f4-7959-48a9-905d-843749c4027b -i td/osd-scrub-repair/0/new.json 2024-08-19T15:25:14.447 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:25:14.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T15:25:14.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDaY8Nm6YGBDRAAPaZWSE3OMPmWa5cP+mrmCA== --osd-uuid 1e09b8f4-7959-48a9-905d-843749c4027b 2024-08-19T15:25:14.487 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:14.485+0000 7ff8905c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:25:14.498 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:14.497+0000 7ff8905c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:25:14.499 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:14.498+0000 7ff8905c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:25:14.500 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:14.498+0000 7ff8905c2640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:25:14.500 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:14.498+0000 7ff8905c2640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:25:14.500 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:14.498+0000 7ff8905c2640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:25:14.581 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:14.580+0000 7ff8905c2640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T15:25:19.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T15:25:19.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:25:19.926 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T15:25:19.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T15:25:19.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:25:20.281 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:25:20.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T15:25:20.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:25:20.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:25:20.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:25:20.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:25:20.308 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:20.307+0000 7f4e60350640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:25:20.339 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:20.338+0000 7f4e60350640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:25:20.342 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:20.340+0000 7f4e60350640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:25:20.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T15:25:20.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:25:20.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:25:20.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:25:20.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:25:20.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:20.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:25:20.550 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:25:20.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:20.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:25:20.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:20.876 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:20.875+0000 7f4e60350640 -1 Falling back to public interface 2024-08-19T15:25:21.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:21.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:21.839 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:25:21.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:25:21.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:21.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:25:22.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:23.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:23.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:23.112 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:25:23.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:25:23.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:23.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:25:23.205 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:23.204+0000 7f4e60350640 -1 osd.0 0 log_to_monitors true 2024-08-19T15:25:23.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:24.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:24.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:24.387 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:25:24.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:25:24.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:24.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:25:24.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:25.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:25.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:25.664 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:25:25.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:25:25.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:25.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:25:26.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:27.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:27.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:27.012 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:25:27.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:25:27.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:27.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:25:27.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:28.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:28.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:28.291 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:25:28.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:25:28.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:28.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:25:28.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:29.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:29.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:29.566 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:25:29.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:25:29.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:29.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:25:29.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:30.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:30.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:30.843 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:25:30.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:25:30.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:30.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:25:31.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:32.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:32.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:32.154 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:25:32.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:25:32.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:32.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:25:32.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:33.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:33.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:33.428 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:25:33.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:25:33.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:33.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:25:33.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:34.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:34.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:34.704 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:25:34.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:25:34.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:34.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:25:35.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:36.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:36.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:36.475 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:25:36.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:25:36.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:36.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:25:37.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:38.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:38.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:38.075 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:25:38.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:25:38.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:38.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:25:38.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:39.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:39.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:39.724 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:25:39.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:25:39.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:39.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:25:40.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:41.112 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:25:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:25:41.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:41.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:25:41.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:42.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:42.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:42.743 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:25:42.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:25:42.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:42.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:25:43.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:44.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:44.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:44.376 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:25:44.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:25:44.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:44.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:25:44.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:45.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:45.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:45.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:25:45.723 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:25:45.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:45.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:25:45.994 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2815821402,v1:127.0.0.1:6803/2815821402] [v2:127.0.0.1:6804/2815821402,v1:127.0.0.1:6805/2815821402] exists,up 1e09b8f4-7959-48a9-905d-843749c4027b 2024-08-19T15:25:45.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:25:45.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:25:45.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:25:45.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-08-19T15:25:45.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-08-19T15:25:45.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:25:45.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:25:45.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T15:25:45.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:25:45.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:25:45.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:25:45.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:25:45.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:25:45.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:25:45.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:25:45.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:25:46.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:25:46.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:25:46.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:25:46.000 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:25:46.000 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:25:46.000 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:25:46.000 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:25:46.000 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:25:46.000 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:25:46.000 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:25:46.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:25:46.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:25:46.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:25:46.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:25:46.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:25:46.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:25:46.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:25:46.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:25:46.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:25:46.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:25:46.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:25:46.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:25:46.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:25:46.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:25:46.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=25ffbc0a-fad2-4d7a-8283-a66df1386cec 2024-08-19T15:25:46.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 25ffbc0a-fad2-4d7a-8283-a66df1386cec' 2024-08-19T15:25:46.002 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 25ffbc0a-fad2-4d7a-8283-a66df1386cec 2024-08-19T15:25:46.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:25:46.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD6Y8NmoV0BARAAbpi1NKOiNXCFEOyNtN2Drw== 2024-08-19T15:25:46.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD6Y8NmoV0BARAAbpi1NKOiNXCFEOyNtN2Drw=="}' 2024-08-19T15:25:46.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 25ffbc0a-fad2-4d7a-8283-a66df1386cec -i td/osd-scrub-repair/1/new.json 2024-08-19T15:25:46.379 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:25:46.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T15:25:46.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD6Y8NmoV0BARAAbpi1NKOiNXCFEOyNtN2Drw== --osd-uuid 25ffbc0a-fad2-4d7a-8283-a66df1386cec 2024-08-19T15:25:46.419 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:46.418+0000 7f8152e3f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:25:46.465 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:46.464+0000 7f8152e3f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:25:46.465 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:46.465+0000 7f8152e3f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:25:46.466 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:46.465+0000 7f8152e3f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:25:46.466 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:46.465+0000 7f8152e3f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:25:46.466 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:46.465+0000 7f8152e3f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:25:46.538 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:46.537+0000 7f8152e3f640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T15:25:51.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T15:25:51.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:25:51.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T15:25:51.739 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T15:25:51.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:25:52.090 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:25:52.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T15:25:52.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:25:52.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:25:52.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:25:52.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:25:52.117 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:52.116+0000 7faad1011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:25:52.119 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:52.118+0000 7faad1011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:25:52.120 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:52.119+0000 7faad1011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:25:52.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T15:25:52.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:25:52.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:25:52.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:25:52.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:25:52.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:52.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:25:52.390 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:25:52.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:52.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:25:52.643 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:52.642+0000 7faad1011640 -1 Falling back to public interface 2024-08-19T15:25:52.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:53.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:53.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:53.657 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:25:53.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:25:53.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:53.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:25:53.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:54.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:54.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:54.930 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:25:54.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:25:54.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:54.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:25:54.986 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:25:54.984+0000 7faad1011640 -1 osd.1 0 log_to_monitors true 2024-08-19T15:25:55.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:56.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:56.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:56.207 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:25:56.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:25:56.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:56.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:25:56.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:57.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:57.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:57.547 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:25:57.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:25:57.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:57.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:25:57.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:25:58.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:25:58.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:25:58.826 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:25:58.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:25:58.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:25:58.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:25:59.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:00.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:00.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:00.122 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:26:00.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:26:00.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:00.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:26:00.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:01.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:01.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:01.415 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:26:01.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:26:01.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:01.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:26:01.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:02.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:02.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:02.783 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:26:02.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:26:02.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:02.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:26:03.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:04.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:04.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:04.069 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:26:04.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:26:04.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:04.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:26:04.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:05.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:05.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:05.421 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:26:05.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:26:05.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:05.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:26:05.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:06.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:06.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:06.802 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:26:06.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:26:06.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:06.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:26:07.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:08.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:08.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:08.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:26:08.287 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:26:08.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:08.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:26:09.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:10.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:10.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:10.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:26:10.299 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:26:10.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:10.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:26:10.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:11.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:11.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:11.905 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:26:11.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:26:11.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:11.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:26:12.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:13.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:13.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:13.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:26:13.518 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:26:13.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:13.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:26:14.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:15.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:15.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:15.122 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:26:15.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:26:15.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:15.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:26:15.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:16.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:16.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:16.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:26:16.669 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:26:16.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:16.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:26:17.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:18.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:18.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:18.095 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:26:18.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:26:18.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:18.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:26:18.399 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3788570272,v1:127.0.0.1:6811/3788570272] [v2:127.0.0.1:6812/3788570272,v1:127.0.0.1:6813/3788570272] exists,up 25ffbc0a-fad2-4d7a-8283-a66df1386cec 2024-08-19T15:26:18.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:26:18.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:26:18.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:26:18.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-08-19T15:26:18.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-08-19T15:26:18.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:26:18.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:26:18.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T15:26:18.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:26:18.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:26:18.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:26:18.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:26:18.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:26:18.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:26:18.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T15:26:18.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T15:26:18.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:26:18.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:26:18.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:26:18.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:26:18.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:26:18.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:26:18.406 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:26:18.406 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:26:18.406 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:26:18.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:26:18.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:26:18.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:26:18.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:26:18.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:26:18.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:26:18.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:26:18.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:26:18.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:26:18.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:26:18.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:26:18.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:26:18.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:26:18.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T15:26:18.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:26:18.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=615f0666-53d2-4851-b3da-49842a1bb490 2024-08-19T15:26:18.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 615f0666-53d2-4851-b3da-49842a1bb490' 2024-08-19T15:26:18.408 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 615f0666-53d2-4851-b3da-49842a1bb490 2024-08-19T15:26:18.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:26:18.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAaZMNmyocZGRAAdM+FBDyjD4xjKs7fInFrlQ== 2024-08-19T15:26:18.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAaZMNmyocZGRAAdM+FBDyjD4xjKs7fInFrlQ=="}' 2024-08-19T15:26:18.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 615f0666-53d2-4851-b3da-49842a1bb490 -i td/osd-scrub-repair/2/new.json 2024-08-19T15:26:18.740 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:26:18.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-08-19T15:26:18.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAaZMNmyocZGRAAdM+FBDyjD4xjKs7fInFrlQ== --osd-uuid 615f0666-53d2-4851-b3da-49842a1bb490 2024-08-19T15:26:18.780 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:26:18.779+0000 7f1e90740640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:26:18.824 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:26:18.823+0000 7f1e90740640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:26:18.825 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:26:18.824+0000 7f1e90740640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:26:18.826 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:26:18.825+0000 7f1e90740640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:26:18.826 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:26:18.825+0000 7f1e90740640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:26:18.826 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:26:18.825+0000 7f1e90740640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:26:18.900 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:26:18.899+0000 7f1e90740640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-08-19T15:26:24.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-08-19T15:26:24.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:26:24.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T15:26:24.344 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T15:26:24.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:26:24.693 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T15:26:24.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T15:26:24.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:26:24.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:26:24.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:26:24.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:26:24.720 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:26:24.719+0000 7f096e018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:26:24.722 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:26:24.721+0000 7f096e018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:26:24.723 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:26:24.722+0000 7f096e018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:26:24.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T15:26:24.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:26:24.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T15:26:24.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:26:24.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:26:24.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:24.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:26:24.965 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:26:24.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:24.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:25.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:25.260 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:26:25.259+0000 7f096e018640 -1 Falling back to public interface 2024-08-19T15:26:26.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:26.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:26.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:26:26.231 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:26:26.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:26.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:26.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:27.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:27.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:27.522 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:26:27.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:26:27.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:27.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:27.598 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:26:27.597+0000 7f096e018640 -1 osd.2 0 log_to_monitors true 2024-08-19T15:26:27.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:28.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:28.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:28.808 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:26:28.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:26:28.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:28.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:29.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:30.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:30.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:30.078 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:26:30.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:26:30.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:30.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:30.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:31.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:31.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:31.391 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:26:31.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:26:31.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:31.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:31.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:32.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:32.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:32.667 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:26:32.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:26:32.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:32.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:33.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:34.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:34.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:34.019 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:26:34.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:26:34.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:34.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:34.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:35.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:35.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:35.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:26:35.297 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:26:35.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:35.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:35.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:36.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:36.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:36.699 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:26:36.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:26:36.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:36.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:36.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:37.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:37.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:37.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:26:37.974 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:26:37.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:37.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:38.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:39.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:39.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:39.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:26:39.251 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:26:39.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:39.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:39.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:40.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:40.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:40.793 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:26:40.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:26:40.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:40.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:41.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:42.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:42.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:42.093 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:26:42.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:26:42.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:42.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:42.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:43.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:43.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:43.504 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:26:43.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:26:43.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:43.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:43.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:44.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:44.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:44.778 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:26:44.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:26:44.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:44.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:45.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:46.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:46.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:46.401 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:26:46.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:26:46.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:46.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:47.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:48.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:48.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:48.140 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:26:48.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:26:48.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:48.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:48.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:49.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:49.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:49.664 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:26:49.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:26:49.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:49.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:49.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:26:50.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:26:50.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:26:50.978 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T15:26:50.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T15:26:50.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:26:50.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:26:51.245 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3137696999,v1:127.0.0.1:6819/3137696999] [v2:127.0.0.1:6820/3137696999,v1:127.0.0.1:6821/3137696999] exists,up 615f0666-53d2-4851-b3da-49842a1bb490 2024-08-19T15:26:51.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:26:51.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:26:51.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:26:51.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-08-19T15:26:51.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T15:26:51.503 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-08-19T15:26:51.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T15:26:51.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T15:26:51.878 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' created 2024-08-19T15:26:51.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:26:52.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T15:26:54.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-08-19T15:26:54.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:26:54.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:26:54.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:26:54.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:26:54.761 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:26:54.761 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:26:54.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:26:54.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:26:54.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:26:54.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:26:54.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:26:54.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:26:54.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:26:54.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:26:54.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:26:55.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:26:55.132 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:26:55.132 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:26:55.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:26:55.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:26:55.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:26:55.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-08-19T15:26:55.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-08-19T15:26:55.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-08-19T15:26:55.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:26:55.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:26:55.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-08-19T15:26:55.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-08-19T15:26:55.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672973' 2024-08-19T15:26:55.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:26:55.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:26:55.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-08-19T15:26:55.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-08-19T15:26:55.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672973 2-64424509445' 2024-08-19T15:26:55.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:26:55.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-08-19T15:26:55.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:26:55.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:26:55.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-08-19T15:26:55.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:26:55.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-08-19T15:26:55.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-08-19T15:26:55.444 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836499 2024-08-19T15:26:55.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:26:55.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-08-19T15:26:55.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:26:56.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:26:56.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:26:56.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836499 2024-08-19T15:26:56.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:26:56.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-08-19T15:26:56.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:26:56.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:26:56.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-08-19T15:26:56.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:26:56.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-08-19T15:26:56.989 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672973 2024-08-19T15:26:56.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-08-19T15:26:56.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:26:57.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-08-19T15:26:57.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:26:57.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-08-19T15:26:57.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:26:57.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:26:57.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-08-19T15:26:57.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:26:57.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-08-19T15:26:57.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-08-19T15:26:57.261 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509445 2024-08-19T15:26:57.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:26:57.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509445 2024-08-19T15:26:57.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:26:57.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:26:57.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:26:57.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T15:26:57.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:26:57.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:26:57.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:26:57.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:26:57.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:26:57.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:26:57.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:26:58.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:26:58.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:26:58.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:26:58.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:26:58.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T15:26:58.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:26:58.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:26:58.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-08-19T15:26:58.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-08-19T15:26:58.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-08-19T15:26:58.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-08-19T15:26:58.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-08-19T15:26:58.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-08-19T15:26:58.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-08-19T15:26:58.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-08-19T15:26:59.492 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'ecpool' created 2024-08-19T15:26:59.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:27:00.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-08-19T15:27:00.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-08-19T15:27:00.993 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 2 allow_ec_overwrites to true 2024-08-19T15:27:01.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-08-19T15:27:01.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:27:01.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:27:01.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:27:01.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:27:01.006 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:27:01.006 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:27:01.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:27:01.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:27:01.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:27:01.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:27:01.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:27:01.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:27:01.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:27:01.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:27:01.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:27:01.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:27:01.369 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:27:01.369 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:27:01.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:27:01.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:27:01.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:27:01.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-08-19T15:27:01.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-08-19T15:27:01.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-08-19T15:27:01.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:27:01.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:27:01.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-08-19T15:27:01.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-08-19T15:27:01.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672975' 2024-08-19T15:27:01.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:27:01.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:27:01.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-08-19T15:27:01.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-08-19T15:27:01.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672975 2-64424509448' 2024-08-19T15:27:01.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:27:01.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-08-19T15:27:01.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:27:01.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:27:01.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-08-19T15:27:01.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:27:01.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-08-19T15:27:01.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-08-19T15:27:01.712 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836502 2024-08-19T15:27:01.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:27:01.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836502 2024-08-19T15:27:01.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:27:02.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:27:02.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:27:03.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-08-19T15:27:03.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:27:03.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-08-19T15:27:03.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:27:03.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:27:03.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-08-19T15:27:03.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:27:03.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-08-19T15:27:03.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-08-19T15:27:03.251 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672975 2024-08-19T15:27:03.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:27:03.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-08-19T15:27:03.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:27:03.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-08-19T15:27:03.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:27:03.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:27:03.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-08-19T15:27:03.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:27:03.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-08-19T15:27:03.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-08-19T15:27:03.522 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509448 2024-08-19T15:27:03.522 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:27:03.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-08-19T15:27:03.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:27:03.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:27:03.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:27:04.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:27:04.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:27:04.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:27:04.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:27:04.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:27:04.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:27:04.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:27:04.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:27:04.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:27:04.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:27:04.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:27:04.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:27:04.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:27:04.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:27:04.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:27:04.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-08-19T15:27:04.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-08-19T15:27:04.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:27:04.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:27:04.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-08-19T15:27:04.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:27:04.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:27:04.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:27:05.190 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:27:05.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:27:05.601 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:27:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:27:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:27:05.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:27:05.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-08-19T15:27:05.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-08-19T15:27:05.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-08-19T15:27:05.873 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-08-19T15:27:05.873 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-19T15:27:06.150 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-08-19T15:27:06.150 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:27:06.150 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-08-19T15:27:06.150 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-08-19T15:27:06.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-08-19T15:27:06.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-08-19T15:27:06.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-08-19T15:27:06.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:27:06.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:27:06.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-08-19T15:27:06.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:27:06.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:27:06.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:27:06.640 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:27:06.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:27:07.009 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:27:07.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:27:07.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:27:07.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:27:07.281 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-08-19T15:27:07.281 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-08-19T15:27:07.281 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-08-19T15:27:07.281 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-08-19T15:27:07.281 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-19T15:27:07.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-08-19T15:27:07.550 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:27:07.550 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-08-19T15:27:07.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-08-19T15:27:07.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-08-19T15:27:07.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-08-19T15:27:07.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-08-19T15:27:07.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-08-19T15:27:07.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:27:07.552 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:27:07.552 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:27:07.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:27:07.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:27:07.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:27:07.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:27:07.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-08-19T15:27:07.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:27:07.860 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:27:07.860 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:27:07.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:27:07.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:27:07.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:27:09.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:27:09.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-08-19T15:27:09.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-08-19T15:27:09.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:27:09.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:27:09.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:27:09.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:27:09.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:27:09.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:27:09.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-08-19T15:27:09.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-08-19T15:27:10.688 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-08-19T15:27:11.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-08-19T15:27:11.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-08-19T15:27:12.460 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-08-19T15:27:12.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-08-19T15:27:12.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-08-19T15:27:13.754 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-08-19T15:27:14.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-08-19T15:27:14.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-08-19T15:27:15.359 INFO:tasks.workunit.client.0.smithi102.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-08-19T15:27:15.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-08-19T15:27:15.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-08-19T15:27:15.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-08-19T15:27:15.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:27:15.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:27:15.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:27:15.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:27:15.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:27:15.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:27:15.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:27:15.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:27:15.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:27:15.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:27:15.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:27:15.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:27:15.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:27:15.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:27:15.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:27:15.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:27:15.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:27:15.878 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:27:15.878 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:27:15.878 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:27:15.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:27:15.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:27:15.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:27:15.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:27:15.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:27:15.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:27:15.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:27:15.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:27:15.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:27:15.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:27:15.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:27:15.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:27:15.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:27:15.881 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:27:15.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:27:15.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:27:15.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:27:15.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:27:15.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:27:15.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:27:15.908 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:15.907+0000 7f3a60015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:27:15.911 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:15.910+0000 7f3a60015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:27:15.912 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:15.911+0000 7f3a60015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:27:16.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:27:16.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:27:16.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:27:16.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:27:16.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:27:16.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:16.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:27:16.162 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:27:16.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:16.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:27:16.429 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:16.428+0000 7f3a60015640 -1 Falling back to public interface 2024-08-19T15:27:16.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:27:17.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:27:17.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:17.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:27:17.474 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:27:17.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:17.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:27:17.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:27:18.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:27:18.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:18.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:27:18.747 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:27:18.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:18.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:27:18.863 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:18.862+0000 7f3a60015640 -1 osd.0 34 log_to_monitors true 2024-08-19T15:27:19.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:27:20.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:27:20.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:20.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:27:20.102 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:27:20.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:20.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:27:20.399 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 41 up_thru 41 down_at 35 last_clean_interval [5,34) [v2:127.0.0.1:6802/3066290601,v1:127.0.0.1:6803/3066290601] [v2:127.0.0.1:6804/3066290601,v1:127.0.0.1:6805/3066290601] exists,up 1e09b8f4-7959-48a9-905d-843749c4027b 2024-08-19T15:27:20.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:27:20.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:27:20.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:27:20.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-08-19T15:27:20.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-08-19T15:27:20.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:27:20.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:27:20.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:27:20.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:27:20.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:27:20.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:27:20.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:27:20.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:27:20.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:27:20.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:27:20.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:27:20.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:27:20.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:27:20.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:27:20.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:27:20.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:27:20.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:27:20.401 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:27:20.401 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:27:20.401 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:27:20.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:27:20.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:27:20.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:27:20.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:27:20.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:27:20.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:27:20.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:27:20.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:27:20.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:27:20.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:27:20.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:27:20.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:27:20.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:27:20.402 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.1 2024-08-19T15:27:20.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:27:20.403 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:27:20.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:27:20.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:27:20.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:27:20.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:27:20.425 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:20.424+0000 7f4c95ba3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:27:20.445 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:20.444+0000 7f4c95ba3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:27:20.446 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:20.445+0000 7f4c95ba3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:27:20.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:27:20.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:27:20.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:27:20.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:27:20.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:27:20.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:20.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:27:20.703 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:27:20.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:20.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:27:20.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:27:20.968 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:20.968+0000 7f4c95ba3640 -1 Falling back to public interface 2024-08-19T15:27:21.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:27:21.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:21.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:27:21.969 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:27:21.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:21.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:27:22.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:27:23.227 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:23.225+0000 7f4c95ba3640 -1 osd.1 36 log_to_monitors true 2024-08-19T15:27:23.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:27:23.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:23.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:27:23.248 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:27:23.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:23.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:27:23.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:27:24.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:27:24.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:24.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:27:24.531 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:27:24.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:24.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:27:24.827 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/877788801,v1:127.0.0.1:6811/877788801] [v2:127.0.0.1:6812/877788801,v1:127.0.0.1:6813/877788801] exists,up 25ffbc0a-fad2-4d7a-8283-a66df1386cec 2024-08-19T15:27:24.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:27:24.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:27:24.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:27:24.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-08-19T15:27:24.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-08-19T15:27:24.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:27:24.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:27:24.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-19T15:27:24.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:27:24.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:27:24.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:27:24.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:27:24.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:27:24.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:27:24.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T15:27:24.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T15:27:24.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:27:24.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:27:24.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:27:24.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:27:24.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:27:24.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:27:24.829 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:27:24.829 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:27:24.829 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:27:24.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:27:24.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:27:24.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:27:24.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:27:24.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:27:24.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:27:24.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:27:24.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:27:24.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:27:24.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:27:24.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:27:24.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T15:27:24.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-19T15:27:24.830 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.2 2024-08-19T15:27:24.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:27:24.830 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-08-19T15:27:24.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-19T15:27:24.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:27:24.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:27:24.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:27:24.853 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:24.852+0000 7fcd73424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:27:24.855 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:24.854+0000 7fcd73424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:27:24.856 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:24.855+0000 7fcd73424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:27:25.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-19T15:27:25.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:27:25.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T15:27:25.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:27:25.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:27:25.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:25.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:27:25.095 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:27:25.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:25.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:27:25.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:27:25.379 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:25.378+0000 7fcd73424640 -1 Falling back to public interface 2024-08-19T15:27:26.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:27:26.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:26.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:27:26.437 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:27:26.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:26.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:27:26.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:27:27.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:27:27.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:27.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:27:27.632 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:27:27.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:27.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:27:27.682 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:27.681+0000 7fcd73424640 -1 osd.2 38 log_to_monitors true 2024-08-19T15:27:27.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:27:28.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:27:28.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:28.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:27:28.907 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:27:28.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:28.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:27:29.288 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 up in weight 1 up_from 47 up_thru 47 down_at 39 last_clean_interval [15,38) [v2:127.0.0.1:6818/4200727898,v1:127.0.0.1:6819/4200727898] [v2:127.0.0.1:6820/4200727898,v1:127.0.0.1:6821/4200727898] exists,up 615f0666-53d2-4851-b3da-49842a1bb490 2024-08-19T15:27:29.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:27:29.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:27:29.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:27:29.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-08-19T15:27:29.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T15:27:29.652 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' removed 2024-08-19T15:27:29.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T15:27:29.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T15:27:30.155 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' created 2024-08-19T15:27:30.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:27:31.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T15:27:32.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-08-19T15:27:32.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:27:32.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:27:32.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:27:32.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:27:32.752 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:27:32.752 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:27:32.752 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:27:32.752 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:27:32.752 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:27:32.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:27:32.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:27:32.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:27:32.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:27:32.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:27:32.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:27:33.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:27:33.121 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:27:33.121 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:27:33.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:27:33.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:27:33.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:27:33.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659140 2024-08-19T15:27:33.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659140 2024-08-19T15:27:33.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659140' 2024-08-19T15:27:33.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:27:33.224 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:27:33.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561027 2024-08-19T15:27:33.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561027 2024-08-19T15:27:33.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659140 1-188978561027' 2024-08-19T15:27:33.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:27:33.326 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:27:33.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462915 2024-08-19T15:27:33.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462915 2024-08-19T15:27:33.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659140 1-188978561027 2-201863462915' 2024-08-19T15:27:33.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:27:33.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-176093659140 2024-08-19T15:27:33.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:27:33.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:27:33.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-176093659140 2024-08-19T15:27:33.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:27:33.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659140 2024-08-19T15:27:33.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 176093659140' 2024-08-19T15:27:33.434 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 176093659140 2024-08-19T15:27:33.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:27:33.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659139 -lt 176093659140 2024-08-19T15:27:33.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:27:34.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:27:34.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:27:34.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659140 -lt 176093659140 2024-08-19T15:27:34.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:27:34.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-188978561027 2024-08-19T15:27:34.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:27:34.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:27:34.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-188978561027 2024-08-19T15:27:34.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:27:34.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561027 2024-08-19T15:27:34.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 188978561027' 2024-08-19T15:27:34.976 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 188978561027 2024-08-19T15:27:34.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:27:35.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561028 -lt 188978561027 2024-08-19T15:27:35.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:27:35.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-201863462915 2024-08-19T15:27:35.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:27:35.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:27:35.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-201863462915 2024-08-19T15:27:35.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:27:35.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462915 2024-08-19T15:27:35.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-08-19T15:27:35.263 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 201863462915 2024-08-19T15:27:35.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:27:35.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462915 -lt 201863462915 2024-08-19T15:27:35.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:27:35.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:27:35.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:27:35.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:27:35.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:27:35.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:27:35.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:27:35.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:27:35.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:27:35.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:27:35.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:27:36.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:27:36.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:27:36.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:27:36.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:27:36.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:27:36.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:27:36.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:27:36.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-08-19T15:27:36.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T15:27:36.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-08-19T15:27:36.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-08-19T15:27:36.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:27:36.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-08-19T15:27:36.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-08-19T15:27:36.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-08-19T15:27:36.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-08-19T15:27:36.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:27:36.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:27:36.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:27:36.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:27:37.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T15:26:59.357457+0000 2024-08-19T15:27:37.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-08-19T15:27:37.272 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-08-19T15:27:37.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-08-19T15:26:59.357457+0000 2024-08-19T15:27:37.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T15:27:37.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:26:59.357457+0000 2024-08-19T15:27:37.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:27:37.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:27:37.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:27:37.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:27:37.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:27:37.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:27:37.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:27:37.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:27:37.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:26:59.357457+0000 '>' 2024-08-19T15:26:59.357457+0000 2024-08-19T15:27:37.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:27:38.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:27:38.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:27:38.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:27:38.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:27:38.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:27:38.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:27:38.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:27:38.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:27:37.761056+0000 '>' 2024-08-19T15:26:59.357457+0000 2024-08-19T15:27:38.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:27:38.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-08-19T15:27:38.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-08-19T15:27:38.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-08-19T15:27:38.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-08-19T15:27:38.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-08-19T15:27:38.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-08-19T15:27:38.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-08-19T15:27:38.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-08-19T15:27:38.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-08-19T15:27:38.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T15:27:38.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:27:38.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:27:38.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:27:38.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:27:38.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:27:38.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:27:38.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:27:38.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:27:39.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:27:39.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:27:39.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:27:39.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:27:39.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:27:39.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:27:39.044 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:27:39.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:27:39.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:27:39.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:27:39.046 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:27:39.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:27:39.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:27:39.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:27:39.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:27:39.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:27:39.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:27:39.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:27:39.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:27:39.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:27:39.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T15:27:39.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T15:27:39.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T15:27:39.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T15:27:39.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:27:39.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:27:39.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:27:39.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:27:39.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:27:39.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:27:39.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:27:39.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:27:39.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:27:39.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:27:39.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:27:39.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:27:39.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:27:39.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:27:39.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:27:39.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:27:39.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:27:39.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:27:39.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:27:39.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:27:39.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:27:39.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:27:39.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:27:39.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:27:39.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:27:39.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:27:39.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:27:39.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:27:39.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T15:27:39.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T15:27:39.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:27:39.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:27:39.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T15:27:39.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T15:27:39.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T15:27:39.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T15:27:39.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T15:27:39.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2024-08-19T15:27:39.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5759: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2024-08-19T15:27:39.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5760: TEST_periodic_scrub_replicated: local poolname=psr_pool 2024-08-19T15:27:39.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5761: TEST_periodic_scrub_replicated: local objname=POBJ 2024-08-19T15:27:39.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5763: TEST_periodic_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-08-19T15:27:39.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T15:27:39.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T15:27:39.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T15:27:39.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T15:27:39.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T15:27:39.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-08-19T15:27:39.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T15:27:39.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:27:39.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:27:39.452 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:27:39.452 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:27:39.452 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:27:39.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:27:39.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-08-19T15:27:39.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T15:27:39.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T15:27:39.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:27:39.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:27:39.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T15:27:39.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T15:27:39.646 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:27:39.647 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:27:39.647 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:27:39.647 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:27:39.647 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:27:39.647 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:27:39.647 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:27:39.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:27:39.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T15:27:39.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T15:27:39.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:27:39.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:27:39.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T15:27:39.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T15:27:39.726 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:27:39.726 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:27:39.726 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:27:39.727 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:27:39.727 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:27:39.727 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:27:39.727 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:27:39.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:27:39.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T15:27:39.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5764: TEST_periodic_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-08-19T15:27:39.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T15:27:39.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T15:27:39.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T15:27:39.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T15:27:39.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T15:27:39.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T15:27:40.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T15:27:40.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:27:40.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:27:40.613 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:27:40.613 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:27:40.614 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:27:40.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:27:40.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:27:40.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:27:40.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5765: TEST_periodic_scrub_replicated: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-08-19T15:27:40.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5766: TEST_periodic_scrub_replicated: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-08-19T15:27:40.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5767: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-08-19T15:27:40.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:27:40.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:27:40.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T15:27:40.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:27:40.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:27:40.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:27:40.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:27:40.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:27:40.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:27:40.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:27:40.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:27:40.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:27:40.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:27:40.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:27:40.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:27:40.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:27:40.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:27:40.649 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:27:40.649 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:27:40.649 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:27:40.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:27:40.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:27:40.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:27:40.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:27:40.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:27:40.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:27:40.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:27:40.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:27:40.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:27:40.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:27:40.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:27:40.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:27:40.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-08-19T15:27:40.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:27:40.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:27:40.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=80d654a6-154c-4359-b99a-d0423d199a10 2024-08-19T15:27:40.651 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 80d654a6-154c-4359-b99a-d0423d199a10 2024-08-19T15:27:40.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 80d654a6-154c-4359-b99a-d0423d199a10' 2024-08-19T15:27:40.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:27:40.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBsZMNmQFe4JxAAqNeNHSRkVz9qK0KrUGVCNg== 2024-08-19T15:27:40.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBsZMNmQFe4JxAAqNeNHSRkVz9qK0KrUGVCNg=="}' 2024-08-19T15:27:40.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 80d654a6-154c-4359-b99a-d0423d199a10 -i td/osd-scrub-repair/0/new.json 2024-08-19T15:27:40.869 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:27:40.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T15:27:40.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQBsZMNmQFe4JxAAqNeNHSRkVz9qK0KrUGVCNg== --osd-uuid 80d654a6-154c-4359-b99a-d0423d199a10 2024-08-19T15:27:40.910 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:40.909+0000 7fee0341e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:27:40.920 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:40.919+0000 7fee0341e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:27:40.921 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:40.920+0000 7fee0341e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:27:40.921 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:40.920+0000 7fee0341e640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:27:40.921 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:40.920+0000 7fee0341e640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:27:40.921 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:40.920+0000 7fee0341e640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:27:41.011 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:41.010+0000 7fee0341e640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T15:27:46.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T15:27:46.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:27:46.498 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T15:27:46.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T15:27:46.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:27:46.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T15:27:46.847 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:27:46.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-08-19T15:27:46.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:27:46.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:27:46.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:27:46.876 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:46.875+0000 7f7e87dad640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:27:46.878 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:46.877+0000 7f7e87dad640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:27:46.879 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:46.878+0000 7f7e87dad640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:27:47.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T15:27:47.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:27:47.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:27:47.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:27:47.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:27:47.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:47.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:27:47.151 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:27:47.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:47.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:27:47.394 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:47.393+0000 7f7e87dad640 -1 Falling back to public interface 2024-08-19T15:27:47.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:27:48.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:27:48.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:48.417 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:27:48.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:27:48.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:48.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:27:48.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:27:49.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:27:49.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:49.692 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:27:49.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:27:49.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:49.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:27:49.720 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:27:49.719+0000 7f7e87dad640 -1 osd.0 0 log_to_monitors true 2024-08-19T15:27:49.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:27:50.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:27:50.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:50.995 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:27:50.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:27:50.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:50.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:27:51.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:27:52.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:27:52.269 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:27:52.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:52.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:27:52.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:52.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:27:52.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:27:53.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:27:53.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:53.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:27:53.598 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:27:53.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:53.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:27:53.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:27:54.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:27:54.876 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:27:54.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:54.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:27:54.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:54.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:27:55.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:27:56.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:27:56.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:56.154 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:27:56.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:27:56.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:56.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:27:56.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:27:57.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:27:57.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:57.426 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:27:57.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:27:57.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:57.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:27:57.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:27:58.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:27:58.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:58.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:27:58.703 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:27:58.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:58.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:27:58.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:27:59.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:27:59.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:27:59.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:27:59.979 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:27:59.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:27:59.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:28:00.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:01.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:01.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:01.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:28:01.388 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:28:01.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:01.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:28:01.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:02.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:02.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:02.890 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:28:02.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:28:02.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:02.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:28:03.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:04.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:04.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:04.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:28:04.342 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:28:04.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:04.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:28:05.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:06.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:06.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:06.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:28:06.020 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:28:06.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:06.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:28:06.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:07.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:07.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:07.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:28:07.429 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:28:07.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:07.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:28:08.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:09.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:09.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:09.167 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:28:09.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:28:09.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:09.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:28:09.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:10.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:10.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:10.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:28:10.853 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:28:10.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:10.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:28:11.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:12.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:12.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:12.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:28:12.131 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:28:12.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:12.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:28:12.401 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2248849484,v1:127.0.0.1:6803/2248849484] [v2:127.0.0.1:6804/2248849484,v1:127.0.0.1:6805/2248849484] exists,up 80d654a6-154c-4359-b99a-d0423d199a10 2024-08-19T15:28:12.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:28:12.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:28:12.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:28:12.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5768: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-08-19T15:28:12.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:28:12.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:28:12.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T15:28:12.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:28:12.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:28:12.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:28:12.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:28:12.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:28:12.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:28:12.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:28:12.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:28:12.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:28:12.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:28:12.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:28:12.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:28:12.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:28:12.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:28:12.407 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:28:12.407 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:28:12.407 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:28:12.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:28:12.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:28:12.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:28:12.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:28:12.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:28:12.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:28:12.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:28:12.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:28:12.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:28:12.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:28:12.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:28:12.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:28:12.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-08-19T15:28:12.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:28:12.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:28:12.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a71975ae-3c21-4a01-9948-d021943543aa 2024-08-19T15:28:12.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a71975ae-3c21-4a01-9948-d021943543aa' 2024-08-19T15:28:12.409 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 a71975ae-3c21-4a01-9948-d021943543aa 2024-08-19T15:28:12.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:28:12.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCMZMNmmiMvGRAA2BYLufV6BGua8xPfW20Yhg== 2024-08-19T15:28:12.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCMZMNmmiMvGRAA2BYLufV6BGua8xPfW20Yhg=="}' 2024-08-19T15:28:12.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a71975ae-3c21-4a01-9948-d021943543aa -i td/osd-scrub-repair/1/new.json 2024-08-19T15:28:12.756 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:28:12.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T15:28:12.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQCMZMNmmiMvGRAA2BYLufV6BGua8xPfW20Yhg== --osd-uuid a71975ae-3c21-4a01-9948-d021943543aa 2024-08-19T15:28:12.799 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:28:12.798+0000 7faa53014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:28:12.824 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:28:12.823+0000 7faa53014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:28:12.825 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:28:12.824+0000 7faa53014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:28:12.826 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:28:12.825+0000 7faa53014640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:28:12.826 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:28:12.825+0000 7faa53014640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:28:12.826 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:28:12.825+0000 7faa53014640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:28:12.899 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:28:12.898+0000 7faa53014640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T15:28:18.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T15:28:18.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:28:18.585 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T15:28:18.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T15:28:18.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:28:18.931 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:28:18.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T15:28:18.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-08-19T15:28:18.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:28:18.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:28:18.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:28:18.962 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:28:18.961+0000 7f820e2d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:28:18.964 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:28:18.963+0000 7f820e2d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:28:18.965 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:28:18.964+0000 7f820e2d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:28:19.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T15:28:19.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:28:19.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:28:19.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:28:19.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:28:19.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:19.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:28:19.203 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:28:19.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:19.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:28:19.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:19.486 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:28:19.485+0000 7f820e2d7640 -1 Falling back to public interface 2024-08-19T15:28:20.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:20.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:20.467 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:28:20.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:28:20.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:20.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:28:20.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:21.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:21.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:21.739 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:28:21.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:28:21.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:21.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:28:21.823 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:28:21.821+0000 7f820e2d7640 -1 osd.1 0 log_to_monitors true 2024-08-19T15:28:22.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:23.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:23.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:23.039 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:28:23.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:28:23.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:23.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:28:23.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:24.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:24.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:24.440 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:28:24.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:28:24.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:24.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:28:24.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:25.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:25.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:25.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:28:25.718 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:28:25.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:25.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:28:25.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:26.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:26.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:26.997 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:28:26.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:28:26.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:26.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:28:27.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:28.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:28.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:28.272 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:28:28.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:28:28.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:28.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:28:28.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:29.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:29.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:29.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:28:29.546 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:28:29.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:29.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:28:29.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:30.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:30.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:30.892 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:28:30.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:28:30.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:30.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:28:31.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:32.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:32.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:32.172 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:28:32.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:28:32.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:32.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:28:32.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:33.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:33.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:33.488 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:28:33.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:28:33.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:33.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:28:33.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:34.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:34.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:34.972 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:28:34.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:28:34.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:34.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:28:35.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:36.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:36.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:36.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:28:36.580 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:28:36.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:36.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:28:37.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:38.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:38.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:38.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:28:38.351 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:28:38.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:38.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:28:38.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:39.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:39.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:39.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:28:39.872 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:28:39.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:39.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:28:40.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:41.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:41.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:41.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:28:41.630 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:28:41.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:41.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:28:41.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:42.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:42.915 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:28:42.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:42.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:28:42.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:42.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:28:43.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:28:44.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:28:44.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:28:44.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:28:44.693 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:28:44.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:28:44.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:28:44.974 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4100243884,v1:127.0.0.1:6811/4100243884] [v2:127.0.0.1:6812/4100243884,v1:127.0.0.1:6813/4100243884] exists,up a71975ae-3c21-4a01-9948-d021943543aa 2024-08-19T15:28:44.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:28:44.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:28:44.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:28:44.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5769: TEST_periodic_scrub_replicated: create_rbd_pool 2024-08-19T15:28:44.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T15:28:45.305 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-08-19T15:28:45.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T15:28:45.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T15:28:45.662 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' created 2024-08-19T15:28:45.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:28:46.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T15:28:48.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5770: TEST_periodic_scrub_replicated: wait_for_clean 2024-08-19T15:28:48.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:28:48.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:28:48.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:28:48.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:28:48.451 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:28:48.451 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:28:48.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:28:48.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:28:48.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:28:48.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:28:48.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:28:48.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:28:48.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:28:48.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:28:48.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:28:48.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:28:48.821 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:28:48.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:28:48.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:28:48.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:28:48.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-08-19T15:28:48.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-08-19T15:28:48.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-08-19T15:28:48.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:28:48.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:28:49.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672965 2024-08-19T15:28:49.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672965 2024-08-19T15:28:49.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-42949672965' 2024-08-19T15:28:49.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:28:49.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-08-19T15:28:49.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:28:49.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:28:49.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-08-19T15:28:49.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:28:49.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-08-19T15:28:49.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-08-19T15:28:49.030 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836493 2024-08-19T15:28:49.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:28:49.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-08-19T15:28:49.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:28:50.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:28:50.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:28:50.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-08-19T15:28:50.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:28:51.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:28:51.573 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:28:51.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-08-19T15:28:51.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:28:51.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672965 2024-08-19T15:28:51.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:28:51.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:28:51.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672965 2024-08-19T15:28:51.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:28:51.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672965 2024-08-19T15:28:51.845 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672965 2024-08-19T15:28:51.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-08-19T15:28:51.845 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:28:52.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672965 2024-08-19T15:28:52.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:28:52.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:28:52.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:28:52.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T15:28:52.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:28:52.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:28:52.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:28:52.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:28:52.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:28:52.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:28:52.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:28:52.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:28:52.709 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:28:52.709 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:28:52.709 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:28:53.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T15:28:53.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:28:53.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:28:53.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5772: TEST_periodic_scrub_replicated: create_pool psr_pool 1 1 2024-08-19T15:28:53.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool 1 1 2024-08-19T15:28:53.428 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'psr_pool' created 2024-08-19T15:28:53.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:28:54.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5773: TEST_periodic_scrub_replicated: wait_for_clean 2024-08-19T15:28:54.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:28:54.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:28:54.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:28:54.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:28:54.442 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:28:54.442 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:28:54.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:28:54.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:28:54.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:28:54.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:28:54.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:28:54.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:28:54.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:28:54.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:28:54.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:28:54.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:28:54.830 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:28:54.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:28:54.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:28:54.831 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:28:54.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-08-19T15:28:54.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-08-19T15:28:54.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-08-19T15:28:54.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:28:54.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:28:55.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672968 2024-08-19T15:28:55.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672968 2024-08-19T15:28:55.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-42949672968' 2024-08-19T15:28:55.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:28:55.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-08-19T15:28:55.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:28:55.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:28:55.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-08-19T15:28:55.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:28:55.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-08-19T15:28:55.043 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836495 2024-08-19T15:28:55.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-08-19T15:28:55.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:28:55.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836495 2024-08-19T15:28:55.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:28:56.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:28:56.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:28:56.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836495 2024-08-19T15:28:56.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:28:57.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:28:57.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:28:57.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836495 2024-08-19T15:28:57.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:28:57.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672968 2024-08-19T15:28:57.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:28:57.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:28:57.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672968 2024-08-19T15:28:57.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:28:57.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672968 2024-08-19T15:28:57.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-08-19T15:28:57.877 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672968 2024-08-19T15:28:57.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:28:58.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672968 2024-08-19T15:28:58.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:28:58.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:28:58.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:28:58.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:28:58.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:28:58.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:28:58.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:28:58.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:28:58.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:28:58.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:28:58.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:28:58.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:28:58.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:28:58.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:28:58.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:28:59.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:28:59.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:28:59.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:28:59.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5775: TEST_periodic_scrub_replicated: local osd=0 2024-08-19T15:28:59.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5776: TEST_periodic_scrub_replicated: add_something td/osd-scrub-repair psr_pool POBJ scrub 2024-08-19T15:28:59.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:28:59.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2024-08-19T15:28:59.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2024-08-19T15:28:59.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=scrub 2024-08-19T15:28:59.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' scrub = noscrub ']' 2024-08-19T15:28:59.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:81: add_something: ceph osd unset noscrub 2024-08-19T15:28:59.462 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is unset 2024-08-19T15:28:59.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:82: add_something: ceph osd unset nodeep-scrub 2024-08-19T15:29:00.006 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is unset 2024-08-19T15:29:00.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:29:00.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:29:00.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-08-19T15:29:00.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: get_primary psr_pool POBJ 2024-08-19T15:29:00.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=psr_pool 2024-08-19T15:29:00.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=POBJ 2024-08-19T15:29:00.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map psr_pool POBJ 2024-08-19T15:29:00.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:29:00.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: local primary=0 2024-08-19T15:29:00.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: get_pg psr_pool POBJ 2024-08-19T15:29:00.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=psr_pool 2024-08-19T15:29:00.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=POBJ 2024-08-19T15:29:00.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map psr_pool POBJ 2024-08-19T15:29:00.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:29:00.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: local pg=2.0 2024-08-19T15:29:00.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5781: TEST_periodic_scrub_replicated: local payload=UVWXYZ 2024-08-19T15:29:00.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5782: TEST_periodic_scrub_replicated: echo UVWXYZ 2024-08-19T15:29:00.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5784: TEST_periodic_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:29:00.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:29:00.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:29:00.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:29:00.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:29:00.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:29:00.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:29:00.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:29:00.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:29:00.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:29:00.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:29:00.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:29:00.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:29:00.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:29:00.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:29:00.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:29:01.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:29:01.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:29:01.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:29:01.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:29:01.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:29:01.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:29:01.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:29:01.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-08-19T15:29:02.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-08-19T15:29:02.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:29:02.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:29:02.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:29:02.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:29:02.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:29:02.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:29:02.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:29:02.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:29:02.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:29:02.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:29:02.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:29:02.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:29:02.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:29:02.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:29:02.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:29:02.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:29:02.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:29:02.932 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:29:02.932 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:29:02.932 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:29:02.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:29:02.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:29:02.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:29:02.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:29:02.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:29:02.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:29:02.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:29:02.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:29:02.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:29:02.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:29:02.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-08-19T15:29:02.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:29:02.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:29:02.934 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:29:02.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-08-19T15:29:02.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:29:02.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:29:02.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:29:02.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:29:02.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:29:02.963 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:29:02.962+0000 7f083bf01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:29:02.965 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:29:02.964+0000 7f083bf01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:29:02.966 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:29:02.966+0000 7f083bf01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:29:03.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:29:03.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:29:03.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:29:03.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:29:03.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:29:03.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:29:03.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:29:03.210 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:29:03.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:29:03.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:29:03.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:29:03.490 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:29:03.489+0000 7f083bf01640 -1 Falling back to public interface 2024-08-19T15:29:04.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:29:04.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:29:04.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:29:04.476 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:29:04.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:29:04.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:29:04.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:29:05.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:29:05.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:29:05.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:29:05.779 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:29:05.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:29:05.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:29:05.822 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:29:05.820+0000 7f083bf01640 -1 osd.0 25 log_to_monitors true 2024-08-19T15:29:06.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:29:07.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:29:07.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:29:07.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:29:07.052 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:29:07.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:29:07.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:29:07.381 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 29 up_thru 29 down_at 26 last_clean_interval [5,25) [v2:127.0.0.1:6802/2680137586,v1:127.0.0.1:6803/2680137586] [v2:127.0.0.1:6804/2680137586,v1:127.0.0.1:6805/2680137586] exists,up 80d654a6-154c-4359-b99a-d0423d199a10 2024-08-19T15:29:07.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:29:07.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:29:07.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:29:07.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:29:07.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:29:07.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:29:07.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:29:07.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:29:07.383 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:29:07.383 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:29:07.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:29:07.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:29:07.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:29:07.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:29:07.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:29:07.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:29:07.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:29:07.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:29:07.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:29:07.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:29:07.744 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:29:07.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:29:07.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:29:07.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:29:07.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-08-19T15:29:07.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-08-19T15:29:07.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586' 2024-08-19T15:29:07.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:29:07.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:29:07.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-08-19T15:29:07.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-08-19T15:29:07.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-42949672971' 2024-08-19T15:29:07.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:29:07.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051586 2024-08-19T15:29:07.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:29:07.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:29:07.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051586 2024-08-19T15:29:07.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:29:07.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-08-19T15:29:07.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-08-19T15:29:07.953 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 124554051586 2024-08-19T15:29:07.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:29:08.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-08-19T15:29:08.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:29:09.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:29:09.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:29:09.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-08-19T15:29:09.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:29:09.494 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-08-19T15:29:09.494 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:29:09.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:29:09.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-08-19T15:29:09.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:29:09.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-08-19T15:29:09.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-08-19T15:29:09.497 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949672971 2024-08-19T15:29:09.497 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:29:09.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672971 2024-08-19T15:29:09.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:29:09.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:29:09.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:29:10.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:29:10.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:29:10.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:29:10.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:29:10.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:29:10.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:29:10.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:29:10.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:29:10.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:29:10.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:29:10.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:29:10.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:29:10.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:29:10.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:29:10.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:29:10.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5787: TEST_periodic_scrub_replicated: set -o pipefail 2024-08-19T15:29:10.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-08-19T15:29:10.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: jq . 2024-08-19T15:29:10.709 INFO:tasks.workunit.client.0.smithi102.stderr:No scrub information available for pg 2.0 2024-08-19T15:29:10.709 INFO:tasks.workunit.client.0.smithi102.stderr:error 2: (2) No such file or directory 2024-08-19T15:29:10.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5789: TEST_periodic_scrub_replicated: set +o pipefail 2024-08-19T15:29:10.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5791: TEST_periodic_scrub_replicated: pg_deep_scrub 2.0 2024-08-19T15:29:10.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=2.0 2024-08-19T15:29:10.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-08-19T15:29:10.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:29:10.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:29:10.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:29:10.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-08-19T15:29:10.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-08-19T15:28:53.385259+0000 2024-08-19T15:29:10.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 2.0 2024-08-19T15:29:11.222 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0 on osd.0 to deep-scrub 2024-08-19T15:29:11.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 2.0 2024-08-19T15:28:53.385259+0000 last_deep_scrub_stamp 2024-08-19T15:29:11.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T15:29:11.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:28:53.385259+0000 2024-08-19T15:29:11.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-08-19T15:29:11.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:29:11.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:29:11.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-08-19T15:29:11.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:29:11.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:29:11.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:29:11.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-08-19T15:29:11.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:28:53.385259+0000 '>' 2024-08-19T15:28:53.385259+0000 2024-08-19T15:29:11.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:29:12.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:29:12.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:29:12.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-08-19T15:29:12.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:29:12.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:29:12.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:29:12.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-08-19T15:29:12.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:28:53.385259+0000 '>' 2024-08-19T15:28:53.385259+0000 2024-08-19T15:29:12.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:29:13.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:29:13.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:29:13.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-08-19T15:29:13.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:29:13.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:29:13.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:29:13.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-08-19T15:29:14.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:28:53.385259+0000 '>' 2024-08-19T15:28:53.385259+0000 2024-08-19T15:29:14.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:29:15.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:29:15.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:29:15.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-08-19T15:29:15.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:29:15.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:29:15.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:29:15.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-08-19T15:29:15.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:28:53.385259+0000 '>' 2024-08-19T15:28:53.385259+0000 2024-08-19T15:29:15.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:29:16.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:29:16.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:29:16.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-08-19T15:29:16.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:29:16.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-08-19T15:29:16.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:29:16.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-08-19T15:29:16.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:29:11.758801+0000 '>' 2024-08-19T15:28:53.385259+0000 2024-08-19T15:29:16.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:29:16.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-08-19T15:29:16.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: jq . 2024-08-19T15:29:16.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: grep -q POBJ 2024-08-19T15:29:16.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5796: TEST_periodic_scrub_replicated: flush_pg_stats 2024-08-19T15:29:16.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:29:16.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:29:16.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:29:16.898 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:29:16.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:29:16.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:29:16.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:29:17.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051589 2024-08-19T15:29:17.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051589 2024-08-19T15:29:17.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589' 2024-08-19T15:29:17.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:29:17.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:29:17.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-08-19T15:29:17.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-08-19T15:29:17.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589 1-42949672974' 2024-08-19T15:29:17.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:29:17.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051589 2024-08-19T15:29:17.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:29:17.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:29:17.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051589 2024-08-19T15:29:17.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:29:17.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051589 2024-08-19T15:29:17.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2024-08-19T15:29:17.110 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 124554051589 2024-08-19T15:29:17.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:29:17.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051588 -lt 124554051589 2024-08-19T15:29:17.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:29:18.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:29:18.377 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:29:18.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051588 -lt 124554051589 2024-08-19T15:29:18.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:29:19.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:29:19.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:29:19.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051589 -lt 124554051589 2024-08-19T15:29:19.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:29:19.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-08-19T15:29:19.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:29:19.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:29:19.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-08-19T15:29:19.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:29:19.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-08-19T15:29:19.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-08-19T15:29:19.934 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672974 2024-08-19T15:29:19.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:29:20.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-08-19T15:29:20.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: get_last_scrub_stamp 2.0 2024-08-19T15:29:20.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:29:20.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:29:20.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:29:20.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:29:20.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: local last_scrub=2024-08-19T15:29:11.758801+0000 2024-08-19T15:29:20.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5799: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-08-19T15:29:20.559 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:29:20.560 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": false, 2024-08-19T15:29:20.560 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:29:20.560 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-12T15:27:40.559552+0000" 2024-08-19T15:29:20.560 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:29:20.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5801: TEST_periodic_scrub_replicated: wait_for_scrub 2.0 2024-08-19T15:29:11.758801+0000 2024-08-19T15:29:20.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T15:29:20.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:29:11.758801+0000 2024-08-19T15:29:20.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:29:20.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:29:20.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:29:20.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:29:20.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:29:20.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:29:20.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:29:20.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:29:20.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:29:11.758801+0000 '>' 2024-08-19T15:29:11.758801+0000 2024-08-19T15:29:20.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:29:21.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:29:21.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:29:21.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:29:21.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:29:21.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:29:21.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:29:21.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:29:22.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:29:11.758801+0000 '>' 2024-08-19T15:29:11.758801+0000 2024-08-19T15:29:22.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:29:23.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:29:23.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:29:23.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:29:23.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:29:23.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:29:23.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:29:23.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:29:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:29:20.832223+0000 '>' 2024-08-19T15:29:11.758801+0000 2024-08-19T15:29:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:29:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5804: TEST_periodic_scrub_replicated: grep -q 'Deep scrub errors, upgrading scrub to deep-scrub' td/osd-scrub-repair/osd.0.log 2024-08-19T15:29:23.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-08-19T15:29:23.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: jq . 2024-08-19T15:29:23.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: grep -q POBJ 2024-08-19T15:29:23.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5810: TEST_periodic_scrub_replicated: ceph osd set nodeep-scrub 2024-08-19T15:29:23.859 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:29:23.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5812: TEST_periodic_scrub_replicated: ceph tell osd.0 get_latest_osdmap 2024-08-19T15:29:23.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5813: TEST_periodic_scrub_replicated: flush_pg_stats 2024-08-19T15:29:23.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:29:23.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:29:24.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:29:24.244 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:29:24.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:29:24.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:29:24.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:29:24.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051591 2024-08-19T15:29:24.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051591 2024-08-19T15:29:24.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591' 2024-08-19T15:29:24.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:29:24.356 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:29:24.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-08-19T15:29:24.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-08-19T15:29:24.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591 1-42949672977' 2024-08-19T15:29:24.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:29:24.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051591 2024-08-19T15:29:24.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:29:24.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:29:24.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051591 2024-08-19T15:29:24.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:29:24.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051591 2024-08-19T15:29:24.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051591' 2024-08-19T15:29:24.462 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 124554051591 2024-08-19T15:29:24.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:29:24.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051591 -lt 124554051591 2024-08-19T15:29:24.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:29:24.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-08-19T15:29:24.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:29:24.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:29:24.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-08-19T15:29:24.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:29:24.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-08-19T15:29:24.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-08-19T15:29:24.733 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672977 2024-08-19T15:29:24.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:29:25.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-08-19T15:29:25.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5814: TEST_periodic_scrub_replicated: sleep 5 2024-08-19T15:29:30.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5817: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-08-19T15:29:30.145 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:29:30.145 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": false, 2024-08-19T15:29:30.145 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:29:30.145 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-12T15:27:50.144769+0000" 2024-08-19T15:29:30.145 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:29:30.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5820: TEST_periodic_scrub_replicated: local found=false 2024-08-19T15:29:30.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: seq 14 -1 0 2024-08-19T15:29:30.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: for i in $(seq 14 -1 0) 2024-08-19T15:29:30.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5823: TEST_periodic_scrub_replicated: sleep 1 2024-08-19T15:29:31.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: grep -q 'Regular scrub skipped due to deep-scrub errors and nodeep-scrub set' td/osd-scrub-repair/osd.0.log 2024-08-19T15:29:31.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: found=true 2024-08-19T15:29:31.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: break 2024-08-19T15:29:31.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5827: TEST_periodic_scrub_replicated: test true = true 2024-08-19T15:29:31.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-08-19T15:29:31.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: jq . 2024-08-19T15:29:31.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: grep -q POBJ 2024-08-19T15:29:31.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5832: TEST_periodic_scrub_replicated: flush_pg_stats 2024-08-19T15:29:31.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:29:31.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:29:31.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:29:31.462 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:29:31.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:29:31.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:29:31.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:29:31.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051594 2024-08-19T15:29:31.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051594 2024-08-19T15:29:31.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051594' 2024-08-19T15:29:31.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:29:31.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:29:31.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-08-19T15:29:31.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-08-19T15:29:31.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051594 1-42949672979' 2024-08-19T15:29:31.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:29:31.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051594 2024-08-19T15:29:31.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:29:31.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:29:31.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051594 2024-08-19T15:29:31.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:29:31.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051594 2024-08-19T15:29:31.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051594' 2024-08-19T15:29:31.673 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 124554051594 2024-08-19T15:29:31.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:29:31.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051592 -lt 124554051594 2024-08-19T15:29:31.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:29:32.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:29:32.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:29:33.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051594 -lt 124554051594 2024-08-19T15:29:33.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:29:33.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-08-19T15:29:33.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:29:33.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:29:33.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-08-19T15:29:33.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:29:33.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-08-19T15:29:33.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-08-19T15:29:33.218 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672979 2024-08-19T15:29:33.218 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:29:33.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672979 2024-08-19T15:29:33.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5834: TEST_periodic_scrub_replicated: pg_schedule_scrub 2.0 2024-08-19T15:29:33.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: pg_schedule_scrub: local pgid=2.0 2024-08-19T15:29:33.484 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: get_last_scrub_stamp 2.0 2024-08-19T15:29:33.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:29:33.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:29:33.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:29:33.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:29:33.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: local last_scrub=2024-08-12T15:27:50.144769+0000 2024-08-19T15:29:33.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: pg_schedule_scrub: ceph pg scrub 2.0 2024-08-19T15:29:33.996 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 2.0 on osd.0 to scrub 2024-08-19T15:29:34.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: pg_schedule_scrub: wait_for_scrub 2.0 2024-08-12T15:27:50.144769+0000 2024-08-19T15:29:34.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-08-19T15:29:34.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-12T15:27:50.144769+0000 2024-08-19T15:29:34.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:29:34.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:29:34.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:29:34.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:29:34.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:29:34.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:29:34.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:29:34.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:29:34.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-12T15:27:50.144769+0000 '>' 2024-08-12T15:27:50.144769+0000 2024-08-19T15:29:34.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:29:35.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:29:35.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:29:35.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:29:35.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:29:35.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:29:35.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:29:35.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:29:35.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-12T15:27:50.144769+0000 '>' 2024-08-12T15:27:50.144769+0000 2024-08-19T15:29:35.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:29:36.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:29:36.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:29:36.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-08-19T15:29:36.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-08-19T15:29:36.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:29:36.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:29:36.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-08-19T15:29:36.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:29:34.786039+0000 '>' 2024-08-12T15:27:50.144769+0000 2024-08-19T15:29:36.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:29:36.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5835: TEST_periodic_scrub_replicated: grep -q 'Regular scrub request, deep-scrub details will be lost' td/osd-scrub-repair/osd.0.log 2024-08-19T15:29:36.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-08-19T15:29:36.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: jq . 2024-08-19T15:29:36.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: grep -qv POBJ 2024-08-19T15:29:36.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T15:29:36.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:29:36.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:29:36.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:29:36.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:29:36.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:29:36.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:29:36.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:29:36.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:29:36.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:29:36.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:29:36.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:29:36.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:29:36.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:29:36.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:29:36.974 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:29:36.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:29:36.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:29:36.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:29:36.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:29:36.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:29:36.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:29:36.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:29:36.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:29:36.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:29:36.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:29:36.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:29:36.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:29:36.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:29:36.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T15:29:36.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T15:29:36.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T15:29:36.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T15:29:36.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:29:36.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:29:36.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:29:36.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:29:36.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:29:36.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:29:36.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:29:36.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:29:36.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:29:36.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:29:36.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:29:36.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:29:36.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:29:36.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:29:36.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:29:36.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:29:36.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:29:36.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:29:36.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:29:36.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:29:37.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:29:37.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:29:37.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:29:37.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:29:37.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:29:37.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:29:37.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:29:37.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:29:37.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T15:29:37.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T15:29:37.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:29:37.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:29:37.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T15:29:37.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T15:29:37.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T15:29:37.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T15:29:37.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T15:29:37.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats td/osd-scrub-repair 2024-08-19T15:29:37.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:654: TEST_repair_stats: local dir=td/osd-scrub-repair 2024-08-19T15:29:37.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:655: TEST_repair_stats: local poolname=testpool 2024-08-19T15:29:37.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_repair_stats: local OSDS=2 2024-08-19T15:29:37.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:657: TEST_repair_stats: local OBJS=30 2024-08-19T15:29:37.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:659: TEST_repair_stats: local REPAIRS=20 2024-08-19T15:29:37.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:662: TEST_repair_stats: run_mon td/osd-scrub-repair a 2024-08-19T15:29:37.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T15:29:37.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T15:29:37.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T15:29:37.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T15:29:37.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T15:29:37.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-08-19T15:29:37.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T15:29:37.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:29:37.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:29:37.379 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:29:37.380 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:29:37.380 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:29:37.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:29:37.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T15:29:37.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T15:29:37.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T15:29:37.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:29:37.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:29:37.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T15:29:37.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T15:29:37.549 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:29:37.549 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:29:37.549 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:29:37.550 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:29:37.550 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:29:37.550 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:29:37.550 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:29:37.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:29:37.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T15:29:37.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T15:29:37.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:29:37.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:29:37.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T15:29:37.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T15:29:37.628 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:29:37.629 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:29:37.629 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:29:37.629 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:29:37.629 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:29:37.630 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:29:37.630 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:29:37.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:29:37.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T15:29:37.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2024-08-19T15:29:37.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T15:29:37.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T15:29:37.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T15:29:37.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T15:29:37.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T15:29:37.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T15:29:38.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T15:29:38.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:29:38.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:29:38.507 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:29:38.507 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:29:38.507 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:29:38.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:29:38.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:29:38.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:29:38.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:665: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T15:29:38.537 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: expr 2 - 1 2024-08-19T15:29:38.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: seq 0 1 2024-08-19T15:29:38.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:29:38.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T15:29:38.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:29:38.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:29:38.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T15:29:38.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:29:38.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:29:38.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:29:38.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:29:38.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:29:38.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:29:38.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:29:38.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:29:38.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:29:38.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:29:38.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:29:38.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:29:38.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:29:38.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:29:38.543 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:29:38.543 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:29:38.543 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:29:38.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:29:38.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:29:38.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:29:38.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:29:38.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:29:38.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:29:38.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:29:38.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:29:38.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:29:38.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:29:38.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:29:38.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:29:38.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T15:29:38.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:29:38.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:29:38.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fd4552b7-34bb-4bf5-8dcc-38050bc71206 2024-08-19T15:29:38.545 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 fd4552b7-34bb-4bf5-8dcc-38050bc71206 2024-08-19T15:29:38.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fd4552b7-34bb-4bf5-8dcc-38050bc71206' 2024-08-19T15:29:38.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:29:38.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDiZMNmU0lmIRAAITcY6ysTAociTwo/a+WZYw== 2024-08-19T15:29:38.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDiZMNmU0lmIRAAITcY6ysTAociTwo/a+WZYw=="}' 2024-08-19T15:29:38.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fd4552b7-34bb-4bf5-8dcc-38050bc71206 -i td/osd-scrub-repair/0/new.json 2024-08-19T15:29:38.747 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:29:38.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T15:29:38.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDiZMNmU0lmIRAAITcY6ysTAociTwo/a+WZYw== --osd-uuid fd4552b7-34bb-4bf5-8dcc-38050bc71206 2024-08-19T15:29:38.788 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:29:38.787+0000 7f7496616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:29:38.790 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:29:38.789+0000 7f7496616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:29:38.791 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:29:38.790+0000 7f7496616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:29:38.791 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:29:38.790+0000 7f7496616640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:29:38.791 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:29:38.790+0000 7f7496616640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:29:38.791 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:29:38.790+0000 7f7496616640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:29:38.881 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:29:38.880+0000 7f7496616640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T15:29:44.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T15:29:44.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:29:44.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T15:29:44.155 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T15:29:44.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:29:44.504 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:29:44.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T15:29:44.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T15:29:44.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:29:44.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:29:44.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:29:44.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:29:44.532+0000 7fe9e6431640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:29:44.535 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:29:44.534+0000 7fe9e6431640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:29:44.535 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:29:44.535+0000 7fe9e6431640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:29:44.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T15:29:44.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:29:44.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:29:44.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:29:44.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:29:44.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:29:44.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:29:44.779 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:29:44.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:29:44.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:29:45.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:29:45.060 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:29:45.059+0000 7fe9e6431640 -1 Falling back to public interface 2024-08-19T15:29:46.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:29:46.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:29:46.046 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:29:46.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:29:46.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:29:46.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:29:46.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:29:47.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:29:47.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:29:47.319 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:29:47.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:29:47.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:29:47.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:29:47.395 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:29:47.394+0000 7fe9e6431640 -1 osd.0 0 log_to_monitors true 2024-08-19T15:29:47.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:29:48.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:29:48.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:29:48.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:29:48.595 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:29:48.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:29:48.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:29:48.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:29:49.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:29:49.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:29:49.871 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:29:49.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:29:49.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:29:49.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:29:50.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:29:51.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:29:51.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:29:51.191 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:29:51.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:29:51.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:29:51.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:29:51.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:29:52.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:29:52.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:29:52.465 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:29:52.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:29:52.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:29:52.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:29:52.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:29:53.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:29:53.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:29:53.744 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:29:53.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:29:53.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:29:53.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:29:54.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:29:55.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:29:55.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:29:55.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:29:55.024 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:29:55.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:29:55.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:29:55.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:29:56.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:29:56.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:29:56.358 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:29:56.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:29:56.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:29:56.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:29:56.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:29:57.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:29:57.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:29:57.633 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:29:57.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:29:57.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:29:57.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:29:57.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:29:58.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:29:58.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:29:58.962 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:29:58.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:29:58.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:29:58.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:29:59.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:00.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:00.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:00.442 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:30:00.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:30:00.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:00.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:30:00.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:01.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:01.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:01.574 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:30:01.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:30:01.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:01.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:30:02.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:03.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:03.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:03.045 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:30:03.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:30:03.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:03.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:30:03.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:04.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:04.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:04.772 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:30:04.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:30:04.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:04.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:30:05.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:06.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:06.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:06.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:30:06.508 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:30:06.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:06.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:30:06.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:07.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:07.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:07.903 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:30:07.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:30:07.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:07.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:30:08.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:09.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:09.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:09.473 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:30:09.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:30:09.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:09.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:30:09.759 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/741058900,v1:127.0.0.1:6803/741058900] [v2:127.0.0.1:6804/741058900,v1:127.0.0.1:6805/741058900] exists,up fd4552b7-34bb-4bf5-8dcc-38050bc71206 2024-08-19T15:30:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:30:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:30:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:30:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:30:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T15:30:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:30:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:30:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T15:30:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:30:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:30:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:30:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:30:09.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:30:09.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:30:09.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:30:09.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:30:09.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:30:09.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:30:09.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:30:09.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:30:09.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:30:09.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:30:09.764 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:30:09.764 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:30:09.764 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:30:09.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:30:09.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:30:09.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:30:09.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:30:09.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:30:09.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:30:09.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:30:09.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:30:09.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:30:09.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:30:09.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:30:09.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:30:09.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T15:30:09.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:30:09.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:30:09.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=402dc8c7-fbab-46ef-b014-a0c5c64af61a 2024-08-19T15:30:09.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 402dc8c7-fbab-46ef-b014-a0c5c64af61a' 2024-08-19T15:30:09.767 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 402dc8c7-fbab-46ef-b014-a0c5c64af61a 2024-08-19T15:30:09.767 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:30:09.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQABZcNmGOWHLhAAdbq1+g+36ru1+iTWsTTZRg== 2024-08-19T15:30:09.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQABZcNmGOWHLhAAdbq1+g+36ru1+iTWsTTZRg=="}' 2024-08-19T15:30:09.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 402dc8c7-fbab-46ef-b014-a0c5c64af61a -i td/osd-scrub-repair/1/new.json 2024-08-19T15:30:10.098 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:30:10.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T15:30:10.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQABZcNmGOWHLhAAdbq1+g+36ru1+iTWsTTZRg== --osd-uuid 402dc8c7-fbab-46ef-b014-a0c5c64af61a 2024-08-19T15:30:10.140 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:30:10.139+0000 7fd28d20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:30:10.182 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:30:10.181+0000 7fd28d20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:30:10.183 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:30:10.182+0000 7fd28d20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:30:10.183 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:30:10.182+0000 7fd28d20d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:30:10.183 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:30:10.182+0000 7fd28d20d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:30:10.183 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:30:10.182+0000 7fd28d20d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:30:10.249 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:30:10.247+0000 7fd28d20d640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T15:30:16.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T15:30:16.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:30:16.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T15:30:16.181 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T15:30:16.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:30:16.528 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:30:16.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T15:30:16.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T15:30:16.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:30:16.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:30:16.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:30:16.556 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:30:16.555+0000 7fab027dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:30:16.558 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:30:16.557+0000 7fab027dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:30:16.559 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:30:16.558+0000 7fab027dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:30:16.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T15:30:16.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:30:16.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:30:16.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:30:16.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:30:16.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:16.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:30:16.797 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:30:16.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:16.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:17.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:17.082 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:30:17.081+0000 7fab027dc640 -1 Falling back to public interface 2024-08-19T15:30:18.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:18.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:18.064 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:30:18.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:30:18.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:18.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:18.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:19.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:19.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:19.339 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:30:19.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:30:19.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:19.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:19.365 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:30:19.364+0000 7fab027dc640 -1 osd.1 0 log_to_monitors true 2024-08-19T15:30:19.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:20.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:20.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:20.666 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:30:20.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:30:20.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:20.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:21.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:22.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:22.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:22.065 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:30:22.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:30:22.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:22.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:22.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:23.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:23.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:23.343 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:30:23.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:30:23.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:23.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:23.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:24.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:24.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:24.615 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:30:24.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:30:24.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:24.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:24.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:25.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:25.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:25.970 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:30:25.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:30:25.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:25.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:26.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:27.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:27.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:27.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:30:27.245 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:30:27.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:27.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:27.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:28.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:28.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:28.521 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:30:28.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:30:28.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:28.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:28.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:29.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:29.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:29.858 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:30:29.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:30:29.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:29.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:30.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:31.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:31.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:31.169 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:30:31.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:30:31.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:31.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:31.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:32.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:32.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:32.490 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:30:32.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:30:32.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:32.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:33.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:34.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:34.036 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:30:34.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:34.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:30:34.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:34.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:34.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:35.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:35.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:35.515 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:30:35.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:30:35.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:35.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:35.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:36.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:36.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:36.943 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:30:36.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:30:36.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:36.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:37.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:38.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:38.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:38.336 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:30:38.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:30:38.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:38.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:38.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:39.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:39.696 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:30:39.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:39.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:30:39.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:39.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:40.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:41.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:41.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:41.156 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:30:41.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:30:41.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:41.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:42.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:30:43.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:30:43.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:30:43.010 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T15:30:43.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T15:30:43.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:30:43.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:30:43.280 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2293283535,v1:127.0.0.1:6811/2293283535] [v2:127.0.0.1:6812/2293283535,v1:127.0.0.1:6813/2293283535] exists,up 402dc8c7-fbab-46ef-b014-a0c5c64af61a 2024-08-19T15:30:43.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:30:43.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:30:43.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:30:43.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:670: TEST_repair_stats: create_pool testpool 1 1 2024-08-19T15:30:43.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-08-19T15:30:43.628 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'testpool' created 2024-08-19T15:30:43.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:30:44.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:671: TEST_repair_stats: ceph osd pool set testpool size 2 2024-08-19T15:30:45.084 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 2 2024-08-19T15:30:45.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_repair_stats: wait_for_clean 2024-08-19T15:30:45.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:30:45.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:30:45.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:30:45.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:30:45.096 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:30:45.096 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:30:45.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:30:45.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:30:45.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:30:45.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:30:45.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:30:45.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:30:45.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:30:45.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:30:45.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:30:45.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:30:45.598 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:30:45.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:30:45.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:30:45.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:30:45.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-08-19T15:30:45.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-08-19T15:30:45.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-08-19T15:30:45.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:30:45.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:30:45.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672965 2024-08-19T15:30:45.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672965 2024-08-19T15:30:45.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672965' 2024-08-19T15:30:45.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:30:45.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-08-19T15:30:45.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:30:45.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:30:45.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-08-19T15:30:45.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:30:45.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-08-19T15:30:45.868 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836497 2024-08-19T15:30:45.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-08-19T15:30:45.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:30:46.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836497 2024-08-19T15:30:46.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:30:47.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:30:47.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:30:47.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-08-19T15:30:47.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:30:47.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672965 2024-08-19T15:30:47.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:30:47.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:30:47.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672965 2024-08-19T15:30:47.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:30:47.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672965 2024-08-19T15:30:47.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-08-19T15:30:47.435 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672965 2024-08-19T15:30:47.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:30:47.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672965 2024-08-19T15:30:47.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:30:47.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:30:47.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:30:48.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:30:48.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:30:48.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:30:48.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:30:48.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:30:48.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:30:48.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:30:48.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:30:48.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-19T15:30:48.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:30:48.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:30:48.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:30:48.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-19T15:30:48.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-19T15:30:48.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T15:30:48.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-19T15:30:48.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T15:30:48.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T15:30:48.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T15:30:48.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:30:48.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:30:48.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:30:48.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:30:48.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:30:48.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:30:48.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:30:48.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:30:48.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:30:48.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:30:48.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:30:49.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:30:49.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:30:49.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:30:49.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:675: TEST_repair_stats: local payload=ABCDEF 2024-08-19T15:30:49.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: echo ABCDEF 2024-08-19T15:30:49.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: seq 1 30 2024-08-19T15:30:49.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:49.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:49.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:49.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:49.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:49.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:49.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:49.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:49.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:49.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:50.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:50.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:50.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:50.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:50.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:50.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:50.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:50.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:50.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:50.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:50.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:50.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:50.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:50.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:50.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:50.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:51.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:51.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:51.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:51.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:51.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:51.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:51.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:51.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:51.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:51.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:51.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:51.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:51.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:51.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:51.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:51.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:51.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:51.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:52.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:52.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:52.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:52.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:52.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:52.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:52.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:52.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:52.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:52.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:52.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:52.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:52.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:52.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:52.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-08-19T15:30:52.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:30:53.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: get_not_primary testpool obj1 2024-08-19T15:30:53.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-08-19T15:30:53.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-08-19T15:30:53.104 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-08-19T15:30:53.104 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T15:30:53.104 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-08-19T15:30:53.105 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-08-19T15:30:53.105 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:30:53.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T15:30:53.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-08-19T15:30:53.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T15:30:53.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: local other=0 2024-08-19T15:30:53.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: get_pg testpool obj1 2024-08-19T15:30:53.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-08-19T15:30:53.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-08-19T15:30:53.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-08-19T15:30:53.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:30:53.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: local pgid=1.0 2024-08-19T15:30:53.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: get_primary testpool obj1 2024-08-19T15:30:53.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T15:30:53.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-08-19T15:30:53.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-08-19T15:30:53.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:30:54.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: local primary=1 2024-08-19T15:30:54.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:688: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:30:54.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:30:54.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:30:54.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:30:54.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:30:54.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:30:54.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:30:54.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:689: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:30:54.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:30:54.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:30:54.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:30:54.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:30:54.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:30:54.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:30:54.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: seq 1 20 2024-08-19T15:30:54.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:30:54.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 1 % 2 2024-08-19T15:30:54.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-08-19T15:30:54.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-08-19T15:30:54.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:30:54.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:30:54.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:30:54.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:30:54.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:30:54.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-08-19T15:30:55.828 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:ff7b1f36:::obj1:head# 2024-08-19T15:30:56.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:30:56.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 2 % 2 2024-08-19T15:30:56.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-08-19T15:30:56.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-08-19T15:30:56.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:30:56.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:30:56.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:30:56.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:30:56.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:30:56.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-08-19T15:30:57.423 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:104778fc:::obj2:head# 2024-08-19T15:30:57.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:30:57.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 3 % 2 2024-08-19T15:30:57.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-08-19T15:30:57.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-08-19T15:30:57.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:30:57.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:30:57.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:30:57.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:30:57.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:30:57.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-08-19T15:30:58.777 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:8dd16f86:::obj3:head# 2024-08-19T15:30:59.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:30:59.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 4 % 2 2024-08-19T15:30:59.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-08-19T15:30:59.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-08-19T15:30:59.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:30:59.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:30:59.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:30:59.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:30:59.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:30:59.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-08-19T15:31:00.038 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:0ee9ae15:::obj4:head# 2024-08-19T15:31:00.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:31:00.552 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 5 % 2 2024-08-19T15:31:00.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-08-19T15:31:00.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-08-19T15:31:00.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:31:00.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:31:00.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:31:00.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:31:00.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:31:00.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-08-19T15:31:01.215 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:c52c9666:::obj5:head# 2024-08-19T15:31:01.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:31:01.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 6 % 2 2024-08-19T15:31:01.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-08-19T15:31:01.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-08-19T15:31:01.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:31:01.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:31:01.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:31:01.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:31:01.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:31:01.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-08-19T15:31:02.392 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:233a42c1:::obj6:head# 2024-08-19T15:31:02.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:31:02.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 7 % 2 2024-08-19T15:31:02.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-08-19T15:31:02.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-08-19T15:31:02.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:31:02.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:31:02.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:31:02.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:31:02.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:31:02.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-08-19T15:31:03.528 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:9a09113a:::obj7:head# 2024-08-19T15:31:04.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:31:04.044 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 8 % 2 2024-08-19T15:31:04.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-08-19T15:31:04.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-08-19T15:31:04.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:31:04.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:31:04.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:31:04.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:31:04.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:31:04.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-08-19T15:31:04.839 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:a557efb1:::obj8:head# 2024-08-19T15:31:05.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:31:05.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 9 % 2 2024-08-19T15:31:05.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-08-19T15:31:05.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-08-19T15:31:05.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:31:05.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:31:05.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:31:05.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:31:05.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:31:05.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-08-19T15:31:06.007 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:fe351e27:::obj9:head# 2024-08-19T15:31:06.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:31:06.522 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 10 % 2 2024-08-19T15:31:06.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-08-19T15:31:06.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-08-19T15:31:06.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:31:06.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:31:06.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:31:06.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:31:06.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:31:06.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-08-19T15:31:07.168 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:d1337354:::obj10:head# 2024-08-19T15:31:07.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:31:07.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 11 % 2 2024-08-19T15:31:07.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-08-19T15:31:07.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-08-19T15:31:07.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:31:07.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:31:07.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:31:07.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:31:07.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:31:07.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-08-19T15:31:08.311 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:eeae2a94:::obj11:head# 2024-08-19T15:31:09.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:31:09.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 12 % 2 2024-08-19T15:31:09.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-08-19T15:31:09.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-08-19T15:31:09.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:31:09.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:31:09.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:31:09.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:31:09.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:31:09.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-08-19T15:31:09.665 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:af5f95cb:::obj12:head# 2024-08-19T15:31:10.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:31:10.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 13 % 2 2024-08-19T15:31:10.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-08-19T15:31:10.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-08-19T15:31:10.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:31:10.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:31:10.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:31:10.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:31:10.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:31:10.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-08-19T15:31:10.968 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:a61ad63e:::obj13:head# 2024-08-19T15:31:11.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:31:11.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 14 % 2 2024-08-19T15:31:11.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-08-19T15:31:11.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-08-19T15:31:11.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:31:11.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:31:11.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:31:11.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:31:11.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:31:11.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-08-19T15:31:12.111 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:e0a44829:::obj14:head# 2024-08-19T15:31:12.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:31:12.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 15 % 2 2024-08-19T15:31:12.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-08-19T15:31:12.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-08-19T15:31:12.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:31:12.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:31:12.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:31:12.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:31:12.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:31:12.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-08-19T15:31:13.256 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:ab946124:::obj15:head# 2024-08-19T15:31:13.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:31:13.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 16 % 2 2024-08-19T15:31:13.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-08-19T15:31:13.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-08-19T15:31:13.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:31:13.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:31:13.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:31:13.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:31:13.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:31:13.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-08-19T15:31:14.357 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:461f8b5e:::obj16:head# 2024-08-19T15:31:14.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:31:14.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 17 % 2 2024-08-19T15:31:14.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-08-19T15:31:14.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-08-19T15:31:14.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:31:14.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:31:14.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:31:14.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:31:14.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:31:14.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-08-19T15:31:15.534 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:24fc6e92:::obj17:head# 2024-08-19T15:31:16.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:31:16.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 18 % 2 2024-08-19T15:31:16.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-08-19T15:31:16.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-08-19T15:31:16.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:31:16.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:31:16.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:31:16.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:31:16.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:31:16.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-08-19T15:31:16.828 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:b9836a99:::obj18:head# 2024-08-19T15:31:17.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:31:17.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 19 % 2 2024-08-19T15:31:17.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-08-19T15:31:17.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-08-19T15:31:17.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:31:17.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:31:17.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:31:17.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:31:17.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:31:17.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-08-19T15:31:17.955 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:00b31b6c:::obj19:head# 2024-08-19T15:31:18.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-08-19T15:31:18.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 20 % 2 2024-08-19T15:31:18.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-08-19T15:31:18.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-08-19T15:31:18.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:31:18.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:31:18.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:31:18.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:31:18.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:31:18.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-08-19T15:31:19.057 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:24b5611b:::obj20:head# 2024-08-19T15:31:19.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:696: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T15:31:19.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:31:19.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:31:19.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:31:19.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:31:19.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:31:19.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:31:19.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:31:19.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:31:19.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:31:19.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:31:19.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:31:19.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:31:19.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:31:19.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:31:19.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:31:19.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:31:19.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:31:19.575 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:31:19.575 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:31:19.575 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:31:19.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:31:19.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:31:19.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:31:19.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:31:19.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:31:19.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:31:19.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:31:19.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:31:19.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:31:19.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:31:19.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T15:31:19.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:31:19.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:31:19.577 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:31:19.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T15:31:19.577 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:31:19.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:31:19.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:31:19.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:31:19.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:31:19.606 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:19.605+0000 7fcda9018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:31:19.608 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:19.607+0000 7fcda9018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:31:19.609 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:19.608+0000 7fcda9018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:31:19.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:31:19.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:31:19.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:31:19.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:31:19.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:31:19.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:31:19.851 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:31:19.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:31:19.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:31:19.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:31:20.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:31:20.133 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:20.132+0000 7fcda9018640 -1 Falling back to public interface 2024-08-19T15:31:21.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:31:21.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:31:21.114 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:31:21.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:31:21.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:31:21.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:31:21.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:31:22.344 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:22.343+0000 7fcda9018640 -1 osd.1 17 log_to_monitors true 2024-08-19T15:31:22.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:31:22.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:31:22.385 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:31:22.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:31:22.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:31:22.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:31:22.740 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 20 up_thru 17 down_at 18 last_clean_interval [10,17) [v2:127.0.0.1:6802/614995962,v1:127.0.0.1:6803/614995962] [v2:127.0.0.1:6804/614995962,v1:127.0.0.1:6805/614995962] exists,up 402dc8c7-fbab-46ef-b014-a0c5c64af61a 2024-08-19T15:31:22.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:31:22.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:31:22.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:31:22.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:697: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T15:31:22.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:31:22.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:31:22.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:31:22.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:31:22.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:31:22.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:31:22.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:31:22.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:31:22.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:31:22.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:31:22.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:31:22.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:31:22.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:31:22.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:31:22.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:31:22.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:31:22.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:31:22.745 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:31:22.745 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:31:22.745 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:31:22.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:31:22.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:31:22.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:31:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:31:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:31:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:31:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:31:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:31:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:31:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:31:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T15:31:22.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:31:22.746 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:31:22.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:31:22.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T15:31:22.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:31:22.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:31:22.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:31:22.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:31:22.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:31:22.771 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:22.770+0000 7fca8e81e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:31:22.806 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:22.806+0000 7fca8e81e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:31:22.808 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:22.807+0000 7fca8e81e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:31:23.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:31:23.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:31:23.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:31:23.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:31:23.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:31:23.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:31:23.019 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:31:23.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:31:23.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:31:23.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:31:23.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:31:23.328 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:23.327+0000 7fca8e81e640 -1 Falling back to public interface 2024-08-19T15:31:24.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:31:24.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:31:24.306 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:31:24.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:31:24.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:31:24.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:31:24.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:31:25.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:31:25.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:31:25.581 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:31:25.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:31:25.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:31:25.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:31:25.611 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:25.609+0000 7fca8e81e640 -1 osd.0 16 log_to_monitors true 2024-08-19T15:31:25.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:31:26.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:31:26.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:31:26.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:31:26.849 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:31:26.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:31:26.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:31:27.117 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 17 last_clean_interval [5,16) [v2:127.0.0.1:6810/2116739689,v1:127.0.0.1:6811/2116739689] [v2:127.0.0.1:6812/2116739689,v1:127.0.0.1:6813/2116739689] exists,up fd4552b7-34bb-4bf5-8dcc-38050bc71206 2024-08-19T15:31:27.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:31:27.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:31:27.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:31:27.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_repair_stats: wait_for_clean 2024-08-19T15:31:27.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:31:27.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:31:27.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:31:27.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:31:27.119 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:31:27.119 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:31:27.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:31:27.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:31:27.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:31:27.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:31:27.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:31:27.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:31:27.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:31:27.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:31:27.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:31:27.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:31:27.480 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:31:27.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:31:27.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:31:27.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:31:27.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247810 2024-08-19T15:31:27.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247810 2024-08-19T15:31:27.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247810' 2024-08-19T15:31:27.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:31:27.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:31:27.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345923 2024-08-19T15:31:27.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345923 2024-08-19T15:31:27.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247810 1-85899345923' 2024-08-19T15:31:27.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:31:27.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247810 2024-08-19T15:31:27.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:31:27.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:31:27.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247810 2024-08-19T15:31:27.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:31:27.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247810 2024-08-19T15:31:27.690 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 98784247810 2024-08-19T15:31:27.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2024-08-19T15:31:27.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:31:27.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 98784247810 2024-08-19T15:31:27.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:31:28.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:31:28.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:31:29.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247810 -lt 98784247810 2024-08-19T15:31:29.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:31:29.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-85899345923 2024-08-19T15:31:29.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:31:29.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:31:29.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-85899345923 2024-08-19T15:31:29.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:31:29.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345923 2024-08-19T15:31:29.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 85899345923' 2024-08-19T15:31:29.232 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 85899345923 2024-08-19T15:31:29.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:31:29.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345923 -lt 85899345923 2024-08-19T15:31:29.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:31:29.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:31:29.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:31:29.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:31:29.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:31:29.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:31:29.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:31:29.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:31:29.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:31:29.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:31:29.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:31:30.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:31:30.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:31:30.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:31:30.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:31:30.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:31:30.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:31:30.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:31:30.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_repair_stats: repair 1.0 2024-08-19T15:31:30.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-08-19T15:31:30.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-08-19T15:31:30.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:31:30.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:31:30.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:31:30.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:31:30.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T15:30:43.577321+0000 2024-08-19T15:31:30.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-08-19T15:31:30.941 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 1.0 on osd.1 to repair 2024-08-19T15:31:30.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-08-19T15:30:43.577321+0000 2024-08-19T15:31:30.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-08-19T15:31:30.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:30:43.577321+0000 2024-08-19T15:31:30.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:31:30.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:31:30.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:31:30.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:31:30.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:31:30.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:31:30.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:31:30.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:31:31.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:30:43.577321+0000 '>' 2024-08-19T15:30:43.577321+0000 2024-08-19T15:31:31.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:31:32.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:31:32.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:31:32.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:31:32.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:31:32.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:31:32.217 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:31:32.217 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:31:32.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:30:43.577321+0000 '>' 2024-08-19T15:30:43.577321+0000 2024-08-19T15:31:32.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:31:33.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:31:33.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:31:33.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:31:33.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:31:33.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:31:33.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:31:33.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:31:33.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:30:43.577321+0000 '>' 2024-08-19T15:30:43.577321+0000 2024-08-19T15:31:33.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:31:34.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:31:34.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:31:34.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:31:34.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:31:34.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:31:34.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:31:34.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:31:35.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:31:31.371470+0000 '>' 2024-08-19T15:30:43.577321+0000 2024-08-19T15:31:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:31:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:701: TEST_repair_stats: wait_for_clean 2024-08-19T15:31:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:31:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:31:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:31:35.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:31:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:31:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:31:35.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:31:35.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:31:35.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:31:35.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:31:35.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:31:35.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:31:35.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:31:35.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:31:35.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:31:35.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:31:35.391 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:31:35.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:31:35.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:31:35.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:31:35.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247812 2024-08-19T15:31:35.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247812 2024-08-19T15:31:35.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247812' 2024-08-19T15:31:35.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:31:35.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:31:35.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345925 2024-08-19T15:31:35.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345925 2024-08-19T15:31:35.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247812 1-85899345925' 2024-08-19T15:31:35.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:31:35.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247812 2024-08-19T15:31:35.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:31:35.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:31:35.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247812 2024-08-19T15:31:35.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:31:35.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247812 2024-08-19T15:31:35.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247812' 2024-08-19T15:31:35.602 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 98784247812 2024-08-19T15:31:35.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:31:35.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247811 -lt 98784247812 2024-08-19T15:31:35.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:31:36.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:31:36.887 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:31:37.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247813 -lt 98784247812 2024-08-19T15:31:37.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:31:37.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-85899345925 2024-08-19T15:31:37.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:31:37.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:31:37.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-85899345925 2024-08-19T15:31:37.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:31:37.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345925 2024-08-19T15:31:37.163 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 85899345925 2024-08-19T15:31:37.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 85899345925' 2024-08-19T15:31:37.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:31:37.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345925 -lt 85899345925 2024-08-19T15:31:37.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:31:37.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:31:37.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:31:37.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:31:37.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:31:37.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:31:37.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:31:37.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:31:37.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:31:37.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:31:37.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:31:38.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:31:38.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:31:38.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:31:38.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:31:38.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:31:38.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:31:38.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:31:38.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:702: TEST_repair_stats: ceph pg dump pgs 2024-08-19T15:31:38.600 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T15:31:38.600 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-08-19T15:31:32.201748+0000 16'30 24:120 [1,0] 1 [1,0] 1 16'30 2024-08-19T15:31:31.371470+0000 16'30 2024-08-19T15:31:31.371470+0000 0 1 periodic scrub scheduled @ 2024-08-20T15:31:31.371470+0000 20 0 2024-08-19T15:31:38.600 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T15:31:38.600 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T15:31:38.600 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:31:38.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:703: TEST_repair_stats: flush_pg_stats 2024-08-19T15:31:38.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:31:38.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:31:38.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:31:38.915 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:31:38.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:31:38.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:31:38.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:31:39.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247814 2024-08-19T15:31:39.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247814 2024-08-19T15:31:39.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247814' 2024-08-19T15:31:39.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:31:39.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:31:39.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345927 2024-08-19T15:31:39.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345927 2024-08-19T15:31:39.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247814 1-85899345927' 2024-08-19T15:31:39.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:31:39.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247814 2024-08-19T15:31:39.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:31:39.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:31:39.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247814 2024-08-19T15:31:39.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:31:39.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247814 2024-08-19T15:31:39.124 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 98784247814 2024-08-19T15:31:39.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247814' 2024-08-19T15:31:39.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:31:39.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247813 -lt 98784247814 2024-08-19T15:31:39.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:31:40.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:31:40.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:31:40.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247814 -lt 98784247814 2024-08-19T15:31:40.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:31:40.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-85899345927 2024-08-19T15:31:40.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:31:40.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:31:40.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-85899345927 2024-08-19T15:31:40.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:31:40.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345927 2024-08-19T15:31:40.677 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 85899345927 2024-08-19T15:31:40.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 85899345927' 2024-08-19T15:31:40.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:31:40.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345927 -lt 85899345927 2024-08-19T15:31:40.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: ceph pg 1.0 query 2024-08-19T15:31:40.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: jq .info.stats.stat_sum 2024-08-19T15:31:41.053 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:31:41.053 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes": 210, 2024-08-19T15:31:41.053 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects": 30, 2024-08-19T15:31:41.053 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_clones": 0, 2024-08-19T15:31:41.053 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_copies": 60, 2024-08-19T15:31:41.053 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing_on_primary": 0, 2024-08-19T15:31:41.053 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing": 0, 2024-08-19T15:31:41.053 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_degraded": 0, 2024-08-19T15:31:41.054 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_misplaced": 0, 2024-08-19T15:31:41.054 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_unfound": 0, 2024-08-19T15:31:41.054 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_dirty": 30, 2024-08-19T15:31:41.054 INFO:tasks.workunit.client.0.smithi102.stdout: "num_whiteouts": 0, 2024-08-19T15:31:41.054 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read": 0, 2024-08-19T15:31:41.054 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read_kb": 0, 2024-08-19T15:31:41.054 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write": 30, 2024-08-19T15:31:41.054 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write_kb": 30, 2024-08-19T15:31:41.054 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrub_errors": 0, 2024-08-19T15:31:41.054 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shallow_scrub_errors": 0, 2024-08-19T15:31:41.054 INFO:tasks.workunit.client.0.smithi102.stdout: "num_deep_scrub_errors": 0, 2024-08-19T15:31:41.054 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_recovered": 20, 2024-08-19T15:31:41.054 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_recovered": 140, 2024-08-19T15:31:41.055 INFO:tasks.workunit.client.0.smithi102.stdout: "num_keys_recovered": 0, 2024-08-19T15:31:41.055 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_omap": 0, 2024-08-19T15:31:41.055 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_hit_set_archive": 0, 2024-08-19T15:31:41.055 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_hit_set_archive": 0, 2024-08-19T15:31:41.055 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush": 0, 2024-08-19T15:31:41.055 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_kb": 0, 2024-08-19T15:31:41.055 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict": 0, 2024-08-19T15:31:41.055 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_kb": 0, 2024-08-19T15:31:41.055 INFO:tasks.workunit.client.0.smithi102.stdout: "num_promote": 0, 2024-08-19T15:31:41.055 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_high": 0, 2024-08-19T15:31:41.055 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_low": 0, 2024-08-19T15:31:41.055 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_some": 0, 2024-08-19T15:31:41.055 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_full": 0, 2024-08-19T15:31:41.056 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_pinned": 0, 2024-08-19T15:31:41.056 INFO:tasks.workunit.client.0.smithi102.stdout: "num_legacy_snapsets": 0, 2024-08-19T15:31:41.056 INFO:tasks.workunit.client.0.smithi102.stdout: "num_large_omap_objects": 0, 2024-08-19T15:31:41.056 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_manifest": 0, 2024-08-19T15:31:41.056 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_bytes": 0, 2024-08-19T15:31:41.056 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_keys": 0, 2024-08-19T15:31:41.056 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_repaired": 20 2024-08-19T15:31:41.056 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:31:41.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: ceph pg 1.0 query 2024-08-19T15:31:41.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2024-08-19T15:31:41.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: COUNT=20 2024-08-19T15:31:41.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: test 20 = 20 2024-08-19T15:31:41.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-08-19T15:31:41.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-08-19T15:31:41.404 INFO:tasks.workunit.client.0.smithi102.stderr:dumped all 2024-08-19T15:31:41.415 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:31:41.415 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:31:41.415 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 20, 2024-08-19T15:31:41.415 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 85899345927, 2024-08-19T15:31:41.415 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 1, 2024-08-19T15:31:41.415 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 1, 2024-08-19T15:31:41.415 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 1, 2024-08-19T15:31:41.415 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 1, 2024-08-19T15:31:41.415 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 104857600, 2024-08-19T15:31:41.416 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 28552, 2024-08-19T15:31:41.416 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 320, 2024-08-19T15:31:41.416 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 4, 2024-08-19T15:31:41.416 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 28219, 2024-08-19T15:31:41.416 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 104829048, 2024-08-19T15:31:41.416 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-08-19T15:31:41.416 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 107374182400, 2024-08-19T15:31:41.416 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 107344945152, 2024-08-19T15:31:41.416 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-08-19T15:31:41.416 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 327680, 2024-08-19T15:31:41.416 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 53726, 2024-08-19T15:31:41.416 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-08-19T15:31:41.416 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-08-19T15:31:41.416 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-08-19T15:31:41.417 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 4398, 2024-08-19T15:31:41.417 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 28896978 2024-08-19T15:31:41.417 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:31:41.417 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [ 2024-08-19T15:31:41.417 INFO:tasks.workunit.client.0.smithi102.stdout: 0 2024-08-19T15:31:41.417 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:31:41.417 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-08-19T15:31:41.417 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-08-19T15:31:41.417 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 10, 2024-08-19T15:31:41.417 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-08-19T15:31:41.417 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-08-19T15:31:41.417 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-08-19T15:31:41.417 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:31:41.417 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-08-19T15:31:41.418 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-08-19T15:31:41.418 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-08-19T15:31:41.418 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-08-19T15:31:41.418 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-08-19T15:31:41.418 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:31:41.418 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [] 2024-08-19T15:31:41.418 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:31:41.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-08-19T15:31:41.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-08-19T15:31:41.667 INFO:tasks.workunit.client.0.smithi102.stderr:dumped all 2024-08-19T15:31:41.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: COUNT=10 2024-08-19T15:31:41.679 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: expr 20 / 2 2024-08-19T15:31:41.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: test 10 = 10 2024-08-19T15:31:41.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-08-19T15:31:41.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-08-19T15:31:41.937 INFO:tasks.workunit.client.0.smithi102.stderr:dumped all 2024-08-19T15:31:41.948 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:31:41.948 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:31:41.948 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 23, 2024-08-19T15:31:41.948 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 98784247814, 2024-08-19T15:31:41.948 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 1, 2024-08-19T15:31:41.948 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 1, 2024-08-19T15:31:41.948 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 1, 2024-08-19T15:31:41.948 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 1, 2024-08-19T15:31:41.948 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 104857600, 2024-08-19T15:31:41.948 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 28552, 2024-08-19T15:31:41.949 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 320, 2024-08-19T15:31:41.949 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 4, 2024-08-19T15:31:41.949 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 28219, 2024-08-19T15:31:41.949 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 104829048, 2024-08-19T15:31:41.949 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-08-19T15:31:41.949 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 107374182400, 2024-08-19T15:31:41.949 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 107344945152, 2024-08-19T15:31:41.949 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-08-19T15:31:41.949 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 327680, 2024-08-19T15:31:41.949 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 53726, 2024-08-19T15:31:41.949 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-08-19T15:31:41.949 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-08-19T15:31:41.949 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-08-19T15:31:41.950 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 4402, 2024-08-19T15:31:41.950 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 28896974 2024-08-19T15:31:41.950 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:31:41.950 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [ 2024-08-19T15:31:41.950 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:31:41.950 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:31:41.950 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-08-19T15:31:41.950 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-08-19T15:31:41.950 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 10, 2024-08-19T15:31:41.950 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-08-19T15:31:41.950 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-08-19T15:31:41.950 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-08-19T15:31:41.950 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:31:41.951 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-08-19T15:31:41.951 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-08-19T15:31:41.951 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-08-19T15:31:41.951 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-08-19T15:31:41.951 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-08-19T15:31:41.951 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:31:41.951 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [] 2024-08-19T15:31:41.951 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:31:41.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-08-19T15:31:41.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-08-19T15:31:42.195 INFO:tasks.workunit.client.0.smithi102.stderr:dumped all 2024-08-19T15:31:42.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: COUNT=10 2024-08-19T15:31:42.207 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: expr 20 / 2 2024-08-19T15:31:42.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: test 10 = 10 2024-08-19T15:31:42.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-08-19T15:31:42.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: jq .pg_map.osd_stats_sum 2024-08-19T15:31:42.457 INFO:tasks.workunit.client.0.smithi102.stderr:dumped all 2024-08-19T15:31:42.468 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:31:42.468 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 0, 2024-08-19T15:31:42.468 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-08-19T15:31:42.468 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 2, 2024-08-19T15:31:42.469 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 2, 2024-08-19T15:31:42.469 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 2, 2024-08-19T15:31:42.469 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 2, 2024-08-19T15:31:42.469 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 209715200, 2024-08-19T15:31:42.469 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 57104, 2024-08-19T15:31:42.469 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 640, 2024-08-19T15:31:42.469 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 8, 2024-08-19T15:31:42.469 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 56439, 2024-08-19T15:31:42.469 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 209658096, 2024-08-19T15:31:42.469 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-08-19T15:31:42.469 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 214748364800, 2024-08-19T15:31:42.469 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 214689890304, 2024-08-19T15:31:42.469 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-08-19T15:31:42.470 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 655360, 2024-08-19T15:31:42.470 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 107452, 2024-08-19T15:31:42.470 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-08-19T15:31:42.470 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-08-19T15:31:42.470 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-08-19T15:31:42.470 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 8800, 2024-08-19T15:31:42.470 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 57793952 2024-08-19T15:31:42.470 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:31:42.470 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [], 2024-08-19T15:31:42.470 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-08-19T15:31:42.470 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-08-19T15:31:42.470 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 20, 2024-08-19T15:31:42.470 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-08-19T15:31:42.471 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-08-19T15:31:42.471 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-08-19T15:31:42.471 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:31:42.471 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-08-19T15:31:42.471 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-08-19T15:31:42.471 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-08-19T15:31:42.471 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-08-19T15:31:42.471 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-08-19T15:31:42.471 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:31:42.471 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [], 2024-08-19T15:31:42.471 INFO:tasks.workunit.client.0.smithi102.stdout: "network_ping_times": [] 2024-08-19T15:31:42.471 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:31:42.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-08-19T15:31:42.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-08-19T15:31:42.719 INFO:tasks.workunit.client.0.smithi102.stderr:dumped all 2024-08-19T15:31:42.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: COUNT=20 2024-08-19T15:31:42.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:720: TEST_repair_stats: test 20 = 20 2024-08-19T15:31:42.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T15:31:42.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:31:42.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:31:42.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:31:42.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:31:42.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:31:42.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:31:42.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:31:42.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:31:42.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:31:42.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:31:42.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:31:42.841 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:31:42.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:31:42.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:31:42.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:31:42.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:31:42.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:31:42.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:31:42.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:31:42.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:31:42.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:31:42.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:31:42.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:31:42.860 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:31:42.860 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:31:42.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:31:42.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:31:42.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:31:42.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T15:31:42.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T15:31:42.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T15:31:42.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T15:31:42.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:31:42.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:31:42.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:31:42.863 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:31:42.863 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:31:42.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:31:42.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:31:42.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:31:42.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:31:42.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:31:42.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:31:42.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:31:42.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:31:42.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:31:42.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:31:42.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:31:42.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:31:42.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:31:42.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:31:42.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:31:42.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:31:42.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:31:42.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:31:42.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:31:42.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:31:42.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:31:42.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:31:42.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:31:42.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T15:31:42.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T15:31:42.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:31:42.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:31:42.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T15:31:42.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T15:31:42.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T15:31:42.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T15:31:42.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T15:31:42.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats_ec td/osd-scrub-repair 2024-08-19T15:31:42.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2024-08-19T15:31:42.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats_ec: local poolname=testpool 2024-08-19T15:31:42.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats_ec: local OSDS=3 2024-08-19T15:31:42.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:727: TEST_repair_stats_ec: local OBJS=30 2024-08-19T15:31:42.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:729: TEST_repair_stats_ec: local REPAIRS=26 2024-08-19T15:31:42.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats_ec: local allow_overwrites=false 2024-08-19T15:31:42.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:733: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2024-08-19T15:31:42.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T15:31:42.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T15:31:42.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T15:31:42.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T15:31:42.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T15:31:42.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-08-19T15:31:43.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T15:31:43.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:31:43.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:31:43.231 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:31:43.231 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:31:43.231 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:31:43.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:31:43.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T15:31:43.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T15:31:43.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T15:31:43.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:31:43.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:31:43.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T15:31:43.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T15:31:43.417 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:31:43.417 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:31:43.417 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:31:43.417 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:31:43.417 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:31:43.418 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:31:43.418 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:31:43.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:31:43.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T15:31:43.497 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T15:31:43.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:31:43.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:31:43.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T15:31:43.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T15:31:43.498 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:31:43.498 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:31:43.498 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:31:43.498 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:31:43.498 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:31:43.498 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:31:43.498 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:31:43.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:31:43.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T15:31:43.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2024-08-19T15:31:43.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T15:31:43.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T15:31:43.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T15:31:43.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T15:31:43.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T15:31:43.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T15:31:44.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T15:31:44.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:31:44.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:31:44.375 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:31:44.375 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:31:44.375 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:31:44.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:31:44.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:31:44.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:31:44.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T15:31:44.406 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: expr 3 - 1 2024-08-19T15:31:44.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: seq 0 2 2024-08-19T15:31:44.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:31:44.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T15:31:44.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:31:44.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:31:44.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T15:31:44.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:31:44.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:31:44.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:31:44.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:31:44.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:31:44.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:31:44.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:31:44.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:31:44.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:31:44.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:31:44.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:31:44.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:31:44.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:31:44.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:31:44.412 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:31:44.412 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:31:44.412 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:31:44.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:31:44.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:31:44.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:31:44.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:31:44.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:31:44.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:31:44.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:31:44.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:31:44.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:31:44.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:31:44.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:31:44.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:31:44.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T15:31:44.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:31:44.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:31:44.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2d9ab22d-cbc4-46e1-8076-0ea490280584 2024-08-19T15:31:44.414 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 2d9ab22d-cbc4-46e1-8076-0ea490280584 2024-08-19T15:31:44.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2d9ab22d-cbc4-46e1-8076-0ea490280584' 2024-08-19T15:31:44.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:31:44.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBgZcNmykiZGRAAvtujpe8Iiq0t4z27Cl3xqQ== 2024-08-19T15:31:44.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBgZcNmykiZGRAAvtujpe8Iiq0t4z27Cl3xqQ=="}' 2024-08-19T15:31:44.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2d9ab22d-cbc4-46e1-8076-0ea490280584 -i td/osd-scrub-repair/0/new.json 2024-08-19T15:31:44.631 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:31:44.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T15:31:44.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBgZcNmykiZGRAAvtujpe8Iiq0t4z27Cl3xqQ== --osd-uuid 2d9ab22d-cbc4-46e1-8076-0ea490280584 2024-08-19T15:31:44.671 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:44.670+0000 7f42d7c12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:31:44.682 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:44.681+0000 7f42d7c12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:31:44.683 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:44.682+0000 7f42d7c12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:31:44.683 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:44.682+0000 7f42d7c12640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:31:44.684 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:44.682+0000 7f42d7c12640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:31:44.684 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:44.682+0000 7f42d7c12640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:31:44.774 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:44.773+0000 7f42d7c12640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T15:31:50.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T15:31:50.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:31:50.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T15:31:50.057 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T15:31:50.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:31:50.404 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:31:50.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T15:31:50.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T15:31:50.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:31:50.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:31:50.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:31:50.433 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:50.431+0000 7fa8a591a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:31:50.447 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:50.446+0000 7fa8a591a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:31:50.448 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:50.448+0000 7fa8a591a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:31:50.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T15:31:50.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:31:50.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:31:50.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:31:50.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:31:50.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:31:50.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:31:50.707 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:31:50.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:31:50.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:31:50.979 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:50.978+0000 7fa8a591a640 -1 Falling back to public interface 2024-08-19T15:31:50.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:31:51.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:31:51.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:31:51.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:31:51.981 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:31:51.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:31:51.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:31:52.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:31:53.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:31:53.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:31:53.257 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:31:53.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:31:53.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:31:53.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:31:53.298 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:31:53.297+0000 7fa8a591a640 -1 osd.0 0 log_to_monitors true 2024-08-19T15:31:53.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:31:54.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:31:54.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:31:54.566 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:31:54.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:31:54.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:31:54.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:31:54.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:31:55.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:31:55.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:31:55.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:31:55.874 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:31:55.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:31:55.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:31:56.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:31:57.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:31:57.246 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:31:57.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:31:57.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:31:57.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:31:57.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:31:57.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:31:58.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:31:58.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:31:58.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:31:58.525 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:31:58.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:31:58.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:31:58.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:31:59.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:31:59.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:31:59.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:31:59.879 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:31:59.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:31:59.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:32:00.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:01.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:01.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:01.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:32:01.209 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:32:01.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:01.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:32:01.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:02.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:02.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:02.484 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:32:02.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:32:02.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:02.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:32:02.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:03.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:03.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:03.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:32:03.759 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:32:03.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:03.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:32:04.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:05.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:05.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:05.047 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:32:05.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:32:05.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:05.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:32:05.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:06.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:06.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:06.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:32:06.713 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:32:06.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:06.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:32:07.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:08.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:08.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:08.140 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:32:08.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:32:08.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:08.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:32:08.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:09.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:09.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:09.426 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:32:09.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:32:09.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:09.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:32:10.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:11.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:11.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:11.154 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:32:11.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:32:11.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:11.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:32:11.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:12.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:12.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:12.717 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:32:12.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:32:12.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:12.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:32:13.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:14.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:14.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:14.016 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:32:14.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:32:14.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:14.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:32:14.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:15.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:15.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:15.420 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:32:15.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:32:15.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:15.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:32:15.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:16.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:16.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:16.694 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T15:32:16.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T15:32:16.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:16.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:32:16.963 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/459583389,v1:127.0.0.1:6803/459583389] [v2:127.0.0.1:6804/459583389,v1:127.0.0.1:6805/459583389] exists,up 2d9ab22d-cbc4-46e1-8076-0ea490280584 2024-08-19T15:32:16.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:32:16.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:32:16.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:32:16.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:32:16.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T15:32:16.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:32:16.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:32:16.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T15:32:16.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:32:16.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:32:16.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:32:16.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:32:16.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:32:16.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:32:16.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:32:16.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:32:16.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:32:16.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:32:16.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:32:16.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:32:16.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:32:16.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:32:16.969 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:32:16.969 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:32:16.969 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:32:16.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:32:16.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:32:16.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:32:16.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:32:16.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:32:16.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:32:16.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:32:16.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:32:16.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:32:16.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:32:16.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:32:16.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:32:16.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T15:32:16.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:32:16.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:32:16.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=24af3092-66bf-4d8b-a4ed-e30b0b55968f 2024-08-19T15:32:16.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 24af3092-66bf-4d8b-a4ed-e30b0b55968f' 2024-08-19T15:32:16.971 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 24af3092-66bf-4d8b-a4ed-e30b0b55968f 2024-08-19T15:32:16.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:32:16.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCAZcNmWBm6OhAAL4P+kwBf6rtXFrpHqz+dIQ== 2024-08-19T15:32:16.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCAZcNmWBm6OhAAL4P+kwBf6rtXFrpHqz+dIQ=="}' 2024-08-19T15:32:16.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 24af3092-66bf-4d8b-a4ed-e30b0b55968f -i td/osd-scrub-repair/1/new.json 2024-08-19T15:32:17.363 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:32:17.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T15:32:17.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCAZcNmWBm6OhAAL4P+kwBf6rtXFrpHqz+dIQ== --osd-uuid 24af3092-66bf-4d8b-a4ed-e30b0b55968f 2024-08-19T15:32:17.405 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:17.404+0000 7fec7b553640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:32:17.449 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:17.448+0000 7fec7b553640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:32:17.450 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:17.449+0000 7fec7b553640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:32:17.451 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:17.450+0000 7fec7b553640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:32:17.451 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:17.450+0000 7fec7b553640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:32:17.451 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:17.450+0000 7fec7b553640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:32:17.524 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:17.523+0000 7fec7b553640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T15:32:23.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T15:32:23.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:32:23.351 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T15:32:23.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T15:32:23.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:32:23.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T15:32:23.696 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:32:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T15:32:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:32:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:32:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:32:23.724 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:23.723+0000 7f1524101640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:32:23.726 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:23.725+0000 7f1524101640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:32:23.727 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:23.726+0000 7f1524101640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:32:23.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T15:32:23.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:32:23.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:32:23.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:32:23.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:32:23.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:23.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:32:23.965 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:32:23.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:23.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:32:24.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:24.260 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:24.260+0000 7f1524101640 -1 Falling back to public interface 2024-08-19T15:32:25.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:25.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:25.230 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:32:25.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:32:25.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:25.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:32:25.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:26.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:26.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:26.506 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:32:26.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:32:26.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:26.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:32:26.730 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:26.728+0000 7f1524101640 -1 osd.1 0 log_to_monitors true 2024-08-19T15:32:26.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:27.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:27.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:27.924 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:32:27.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:32:27.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:27.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:32:28.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:29.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:29.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:29.202 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:32:29.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:32:29.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:29.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:32:29.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:30.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:30.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:30.533 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:32:30.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:32:30.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:30.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:32:30.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:31.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:31.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:31.893 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:32:31.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:32:31.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:31.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:32:32.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:33.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:33.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:33.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:32:33.170 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:32:33.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:33.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:32:33.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:34.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:34.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:34.612 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:32:34.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:32:34.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:34.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:32:34.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:35.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:35.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:35.893 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:32:35.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:32:35.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:35.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:32:36.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:37.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:37.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:37.166 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:32:37.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:32:37.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:37.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:32:37.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:38.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:38.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:38.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:32:38.502 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:32:38.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:38.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:32:39.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:40.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:40.448 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:32:40.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:40.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:32:40.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:40.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:32:41.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:42.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:42.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:42.284 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:32:42.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:32:42.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:42.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:32:42.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:43.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:43.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:43.719 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:32:43.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:32:43.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:43.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:32:44.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:45.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:45.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:45.414 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:32:45.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:32:45.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:45.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:32:45.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:46.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:46.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:46.924 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:32:46.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:32:46.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:46.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:32:47.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:48.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:48.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:48.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:32:48.502 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:32:48.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:48.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:32:48.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:49.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:49.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:49.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:32:50.000 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:32:50.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:50.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:32:50.310 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/309511507,v1:127.0.0.1:6811/309511507] [v2:127.0.0.1:6812/309511507,v1:127.0.0.1:6813/309511507] exists,up 24af3092-66bf-4d8b-a4ed-e30b0b55968f 2024-08-19T15:32:50.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:32:50.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:32:50.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:32:50.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:32:50.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T15:32:50.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:32:50.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:32:50.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T15:32:50.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:32:50.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:32:50.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:32:50.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:32:50.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:32:50.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:32:50.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T15:32:50.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T15:32:50.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:32:50.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:32:50.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:32:50.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:32:50.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:32:50.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:32:50.316 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:32:50.316 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:32:50.316 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:32:50.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:32:50.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:32:50.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:32:50.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:32:50.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:32:50.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:32:50.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:32:50.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:32:50.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:32:50.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:32:50.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:32:50.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:32:50.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T15:32:50.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T15:32:50.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:32:50.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9f0ee062-9891-468a-8c16-c3ab403fe52e 2024-08-19T15:32:50.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9f0ee062-9891-468a-8c16-c3ab403fe52e' 2024-08-19T15:32:50.317 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 9f0ee062-9891-468a-8c16-c3ab403fe52e 2024-08-19T15:32:50.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:32:50.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCiZcNmgjSTExAADLENRwcb8pDB27ujUBwuow== 2024-08-19T15:32:50.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCiZcNmgjSTExAADLENRwcb8pDB27ujUBwuow=="}' 2024-08-19T15:32:50.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9f0ee062-9891-468a-8c16-c3ab403fe52e -i td/osd-scrub-repair/2/new.json 2024-08-19T15:32:50.666 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:32:50.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-08-19T15:32:50.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCiZcNmgjSTExAADLENRwcb8pDB27ujUBwuow== --osd-uuid 9f0ee062-9891-468a-8c16-c3ab403fe52e 2024-08-19T15:32:50.707 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:50.706+0000 7ff213e1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:32:50.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:50.741+0000 7ff213e1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:32:50.743 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:50.742+0000 7ff213e1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:32:50.743 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:50.742+0000 7ff213e1f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:32:50.744 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:50.742+0000 7ff213e1f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:32:50.744 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:50.742+0000 7ff213e1f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:32:50.817 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:50.816+0000 7ff213e1f640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-08-19T15:32:56.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-08-19T15:32:56.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:32:56.544 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T15:32:56.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T15:32:56.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:32:56.890 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T15:32:56.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T15:32:56.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:32:56.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T15:32:56.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:32:56.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:32:56.918 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:56.917+0000 7f0999843640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:32:56.920 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:56.919+0000 7f0999843640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:32:56.921 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:56.920+0000 7f0999843640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:32:57.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T15:32:57.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:32:57.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T15:32:57.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:32:57.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:32:57.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:57.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:32:57.156 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:32:57.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:57.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:32:57.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:57.445 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:57.444+0000 7f0999843640 -1 Falling back to public interface 2024-08-19T15:32:58.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:58.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:58.422 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:32:58.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:32:58.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:58.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:32:58.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:32:59.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:32:59.700 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:32:59.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:32:59.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:32:59.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:32:59.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:32:59.757 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:32:59.756+0000 7f0999843640 -1 osd.2 0 log_to_monitors true 2024-08-19T15:32:59.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:33:00.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:33:00.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:33:00.979 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:33:00.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:33:00.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:33:00.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:33:01.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:33:02.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:33:02.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:33:02.273 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:33:02.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:33:02.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:33:02.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:33:02.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:33:03.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:33:03.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:33:03.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:33:03.554 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:33:03.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:33:03.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:33:03.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:33:04.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:33:04.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:33:04.832 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:33:04.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:33:04.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:33:04.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:33:05.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:33:06.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:33:06.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:33:06.241 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:33:06.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:33:06.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:33:06.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:33:06.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:33:07.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:33:07.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:33:07.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:33:07.581 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:33:07.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:33:07.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:33:07.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:33:08.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:33:08.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:33:08.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:33:08.858 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:33:08.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:33:08.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:33:09.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:33:10.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:33:10.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:33:10.169 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:33:10.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:33:10.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:33:10.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:33:10.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:33:11.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:33:11.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:33:11.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:33:11.449 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:33:11.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:33:11.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:33:11.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:33:12.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:33:12.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:33:12.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:33:12.739 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:33:12.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:33:12.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:33:13.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:33:14.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:33:14.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:33:14.345 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:33:14.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:33:14.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:33:14.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:33:15.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:33:16.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:33:16.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:33:16.004 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:33:16.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:33:16.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:33:16.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:33:16.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:33:17.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:33:17.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:33:17.465 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:33:17.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:33:17.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:33:17.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:33:18.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:33:19.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:33:19.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:33:19.033 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:33:19.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:33:19.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:33:19.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:33:19.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:33:20.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:33:20.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:33:20.672 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:33:20.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:33:20.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:33:20.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:33:21.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:33:22.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:33:22.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:33:22.022 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:33:22.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:33:22.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:33:22.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:33:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:33:23.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:33:23.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:33:23.301 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T15:33:23.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T15:33:23.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:33:23.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:33:23.576 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3770081873,v1:127.0.0.1:6819/3770081873] [v2:127.0.0.1:6820/3770081873,v1:127.0.0.1:6821/3770081873] exists,up 9f0ee062-9891-468a-8c16-c3ab403fe52e 2024-08-19T15:33:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:33:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:33:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:33:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:742: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2024-08-19T15:33:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=testpool 2024-08-19T15:33:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-08-19T15:33:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-08-19T15:33:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-08-19T15:33:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-08-19T15:33:23.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2024-08-19T15:33:23.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2024-08-19T15:33:24.671 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'testpool' created 2024-08-19T15:33:24.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:33:25.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-08-19T15:33:25.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-08-19T15:33:25.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:33:25.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:33:25.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:33:25.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:33:25.685 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:33:25.685 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:33:25.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:33:25.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:33:25.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:33:25.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:33:25.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:33:25.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:33:25.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:33:25.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:33:25.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:33:26.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:33:26.089 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:33:26.089 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:33:26.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:33:26.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:33:26.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:33:26.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-08-19T15:33:26.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-08-19T15:33:26.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-08-19T15:33:26.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:33:26.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:33:26.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-08-19T15:33:26.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-08-19T15:33:26.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672974' 2024-08-19T15:33:26.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:33:26.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:33:26.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-08-19T15:33:26.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-08-19T15:33:26.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672974 2-64424509445' 2024-08-19T15:33:26.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:33:26.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-08-19T15:33:26.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:33:26.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:33:26.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-08-19T15:33:26.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:33:26.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-08-19T15:33:26.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-08-19T15:33:26.405 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836500 2024-08-19T15:33:26.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:33:26.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-08-19T15:33:26.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:33:26.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-08-19T15:33:26.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:33:26.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:33:26.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-08-19T15:33:26.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:33:26.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-08-19T15:33:26.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-08-19T15:33:26.678 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672974 2024-08-19T15:33:26.678 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:33:26.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672974 2024-08-19T15:33:26.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:33:27.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:33:27.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:33:28.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672974 2024-08-19T15:33:28.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:33:29.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:33:29.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:33:29.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-08-19T15:33:29.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:33:29.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-08-19T15:33:29.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:33:29.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:33:29.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-08-19T15:33:29.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:33:29.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-08-19T15:33:29.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-08-19T15:33:29.526 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509445 2024-08-19T15:33:29.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:33:29.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509445 2024-08-19T15:33:29.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:33:29.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:33:29.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:33:30.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:33:30.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:33:30.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:33:30.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:33:30.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:33:30.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:33:30.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:33:30.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:33:30.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:33:30.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:33:30.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:33:30.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:33:30.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:33:30.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:33:30.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:33:30.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-08-19T15:33:30.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats_ec: local payload=ABCDEF 2024-08-19T15:33:30.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: echo ABCDEF 2024-08-19T15:33:30.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: seq 1 30 2024-08-19T15:33:30.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:30.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:30.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:30.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:31.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:31.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:31.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:31.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:31.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:31.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:31.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:31.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:32.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:32.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:32.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:32.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:32.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:32.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:32.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:32.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:32.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:32.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:33.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:33.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:33.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:33.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:33.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:33.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:33.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:33.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:33.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:33.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:34.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:34.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:34.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:34.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:34.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:34.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:34.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:34.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:35.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:35.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:35.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:35.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:35.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:35.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:35.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:35.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:36.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:36.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:36.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:36.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:36.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:36.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:36.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:36.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:37.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:37.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:37.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-08-19T15:33:37.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-08-19T15:33:37.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: get_not_primary testpool obj1 2024-08-19T15:33:37.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-08-19T15:33:37.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-08-19T15:33:37.402 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-08-19T15:33:37.403 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T15:33:37.403 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-08-19T15:33:37.406 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-08-19T15:33:37.406 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:33:37.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T15:33:37.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-08-19T15:33:37.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T15:33:37.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: local other=0 2024-08-19T15:33:37.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: get_pg testpool obj1 2024-08-19T15:33:37.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-08-19T15:33:37.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-08-19T15:33:37.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-08-19T15:33:37.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:33:38.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: local pgid=1.0 2024-08-19T15:33:38.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: get_primary testpool obj1 2024-08-19T15:33:38.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-08-19T15:33:38.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-08-19T15:33:38.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-08-19T15:33:38.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:33:38.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: local primary=1 2024-08-19T15:33:38.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:758: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:33:38.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:33:38.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:33:38.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:33:38.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:33:38.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:33:38.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:33:38.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:759: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2024-08-19T15:33:38.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:33:38.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:33:38.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:33:38.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:33:38.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:33:40.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:33:40.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: seq 1 26 2024-08-19T15:33:40.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:33:40.100 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 1 % 2 2024-08-19T15:33:40.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-08-19T15:33:40.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-08-19T15:33:40.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:33:40.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:33:40.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:33:40.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:33:40.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:33:40.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-08-19T15:33:41.317 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#1:ff7b1f36:::obj1:head# 2024-08-19T15:33:41.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:33:41.836 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 2 % 2 2024-08-19T15:33:41.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-08-19T15:33:41.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-08-19T15:33:41.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:33:41.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:33:41.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:33:41.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:33:41.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:33:41.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-08-19T15:33:43.042 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#1:104778fc:::obj2:head# 2024-08-19T15:33:43.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:33:43.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 3 % 2 2024-08-19T15:33:43.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-08-19T15:33:43.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-08-19T15:33:43.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:33:43.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:33:43.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:33:43.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:33:43.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:33:43.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-08-19T15:33:44.313 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#1:8dd16f86:::obj3:head# 2024-08-19T15:33:44.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:33:44.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 4 % 2 2024-08-19T15:33:44.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-08-19T15:33:44.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-08-19T15:33:44.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:33:44.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:33:44.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:33:44.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:33:44.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:33:44.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-08-19T15:33:45.566 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#1:0ee9ae15:::obj4:head# 2024-08-19T15:33:46.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:33:46.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 5 % 2 2024-08-19T15:33:46.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-08-19T15:33:46.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-08-19T15:33:46.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:33:46.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:33:46.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:33:46.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:33:46.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:33:46.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-08-19T15:33:46.802 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#1:c52c9666:::obj5:head# 2024-08-19T15:33:47.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:33:47.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 6 % 2 2024-08-19T15:33:47.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-08-19T15:33:47.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-08-19T15:33:47.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:33:47.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:33:47.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:33:47.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:33:47.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:33:47.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-08-19T15:33:47.995 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#1:233a42c1:::obj6:head# 2024-08-19T15:33:48.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:33:48.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 7 % 2 2024-08-19T15:33:48.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-08-19T15:33:48.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-08-19T15:33:48.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:33:48.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:33:48.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:33:48.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:33:48.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:33:48.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-08-19T15:33:49.206 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#1:9a09113a:::obj7:head# 2024-08-19T15:33:49.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:33:49.723 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 8 % 2 2024-08-19T15:33:49.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-08-19T15:33:49.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-08-19T15:33:49.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:33:49.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:33:49.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:33:49.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:33:49.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:33:49.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-08-19T15:33:50.407 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#1:a557efb1:::obj8:head# 2024-08-19T15:33:50.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:33:50.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 9 % 2 2024-08-19T15:33:50.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-08-19T15:33:50.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-08-19T15:33:50.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:33:50.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:33:50.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:33:50.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:33:50.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:33:50.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-08-19T15:33:51.569 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#1:fe351e27:::obj9:head# 2024-08-19T15:33:52.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:33:52.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 10 % 2 2024-08-19T15:33:52.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-08-19T15:33:52.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-08-19T15:33:52.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:33:52.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:33:52.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:33:52.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:33:52.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:33:52.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-08-19T15:33:52.838 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#1:d1337354:::obj10:head# 2024-08-19T15:33:53.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:33:53.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 11 % 2 2024-08-19T15:33:53.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-08-19T15:33:53.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-08-19T15:33:53.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:33:53.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:33:53.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:33:53.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:33:53.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:33:53.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-08-19T15:33:54.031 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#1:eeae2a94:::obj11:head# 2024-08-19T15:33:54.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:33:54.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 12 % 2 2024-08-19T15:33:54.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-08-19T15:33:54.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-08-19T15:33:54.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:33:54.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:33:54.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:33:54.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:33:54.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:33:54.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-08-19T15:33:55.451 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#1:af5f95cb:::obj12:head# 2024-08-19T15:33:55.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:33:55.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 13 % 2 2024-08-19T15:33:55.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-08-19T15:33:55.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-08-19T15:33:55.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:33:55.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:33:55.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:33:55.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:33:55.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:33:55.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-08-19T15:33:56.738 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#1:a61ad63e:::obj13:head# 2024-08-19T15:33:57.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:33:57.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 14 % 2 2024-08-19T15:33:57.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-08-19T15:33:57.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-08-19T15:33:57.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:33:57.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:33:57.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:33:57.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:33:57.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:33:57.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-08-19T15:33:57.873 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#1:e0a44829:::obj14:head# 2024-08-19T15:33:58.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:33:58.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 15 % 2 2024-08-19T15:33:58.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-08-19T15:33:58.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-08-19T15:33:58.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:33:58.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:33:58.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:33:58.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:33:58.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:33:58.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-08-19T15:33:58.959 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#1:ab946124:::obj15:head# 2024-08-19T15:33:59.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:33:59.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 16 % 2 2024-08-19T15:33:59.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-08-19T15:33:59.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-08-19T15:33:59.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:33:59.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:33:59.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:33:59.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:33:59.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:33:59.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-08-19T15:34:00.060 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#1:461f8b5e:::obj16:head# 2024-08-19T15:34:00.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:34:00.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 17 % 2 2024-08-19T15:34:00.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-08-19T15:34:00.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-08-19T15:34:00.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:34:00.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:34:00.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:34:00.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:34:00.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:34:00.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-08-19T15:34:01.212 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#1:24fc6e92:::obj17:head# 2024-08-19T15:34:01.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:34:01.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 18 % 2 2024-08-19T15:34:01.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-08-19T15:34:01.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-08-19T15:34:01.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:34:01.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:34:01.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:34:01.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:34:01.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:34:01.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-08-19T15:34:02.371 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#1:b9836a99:::obj18:head# 2024-08-19T15:34:02.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:34:02.886 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 19 % 2 2024-08-19T15:34:02.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-08-19T15:34:02.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-08-19T15:34:02.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:34:02.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:34:02.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:34:02.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:34:02.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:34:02.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-08-19T15:34:03.441 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#1:00b31b6c:::obj19:head# 2024-08-19T15:34:03.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:34:03.959 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 20 % 2 2024-08-19T15:34:03.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-08-19T15:34:03.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-08-19T15:34:03.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:34:03.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:34:03.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:34:03.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:34:03.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:34:03.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-08-19T15:34:04.702 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#1:24b5611b:::obj20:head# 2024-08-19T15:34:05.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:34:05.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 21 % 2 2024-08-19T15:34:05.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-08-19T15:34:05.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2024-08-19T15:34:05.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:34:05.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:34:05.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:34:05.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:34:05.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:34:05.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2024-08-19T15:34:05.870 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#1:1243fe1a:::obj21:head# 2024-08-19T15:34:06.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:34:06.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 22 % 2 2024-08-19T15:34:06.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-08-19T15:34:06.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2024-08-19T15:34:06.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:34:06.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:34:06.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:34:06.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:34:06.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:34:06.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2024-08-19T15:34:07.072 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#1:c0c49def:::obj22:head# 2024-08-19T15:34:07.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:34:07.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 23 % 2 2024-08-19T15:34:07.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-08-19T15:34:07.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2024-08-19T15:34:07.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:34:07.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:34:07.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:34:07.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:34:07.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:34:07.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2024-08-19T15:34:08.191 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#1:242c40e0:::obj23:head# 2024-08-19T15:34:08.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:34:08.709 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 24 % 2 2024-08-19T15:34:08.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-08-19T15:34:08.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2024-08-19T15:34:08.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:34:08.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:34:08.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:34:08.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:34:08.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:34:08.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2024-08-19T15:34:09.335 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#1:7eacede5:::obj24:head# 2024-08-19T15:34:09.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:34:09.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 25 % 2 2024-08-19T15:34:09.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-08-19T15:34:09.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2024-08-19T15:34:09.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:34:09.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:34:09.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-19T15:34:09.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:34:09.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:34:09.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2024-08-19T15:34:10.624 INFO:tasks.workunit.client.0.smithi102.stdout:remove 0#1:7109478f:::obj25:head# 2024-08-19T15:34:11.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-08-19T15:34:11.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 26 % 2 2024-08-19T15:34:11.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-08-19T15:34:11.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2024-08-19T15:34:11.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:34:11.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:34:11.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:34:11.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:34:11.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:34:11.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2024-08-19T15:34:11.768 INFO:tasks.workunit.client.0.smithi102.stdout:remove 1#1:f3c2593a:::obj26:head# 2024-08-19T15:34:12.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T15:34:12.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:34:12.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:34:12.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:34:12.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:34:12.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:34:12.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:34:12.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:34:12.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:34:12.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:34:12.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:34:12.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:34:12.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:34:12.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:34:12.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:34:12.287 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:34:12.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:34:12.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:34:12.288 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:34:12.288 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:34:12.288 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:34:12.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:34:12.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:34:12.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:34:12.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:34:12.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:34:12.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:34:12.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:34:12.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:34:12.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:34:12.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:34:12.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T15:34:12.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:34:12.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:34:12.289 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:34:12.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T15:34:12.290 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-08-19T15:34:12.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:34:12.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:34:12.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:34:12.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:34:12.318 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:12.317+0000 7f64cc7dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:34:12.320 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:12.319+0000 7f64cc7dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:34:12.321 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:12.320+0000 7f64cc7dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:34:12.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:34:12.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:34:12.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:34:12.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:34:12.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:34:12.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:34:12.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:34:12.591 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:34:12.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:34:12.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:34:12.843 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:12.842+0000 7f64cc7dc640 -1 Falling back to public interface 2024-08-19T15:34:12.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:34:13.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:34:13.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:34:13.864 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:34:13.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:34:13.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:34:13.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:34:14.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:34:15.106 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:15.105+0000 7f64cc7dc640 -1 osd.1 23 log_to_monitors true 2024-08-19T15:34:15.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:34:15.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:34:15.138 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:34:15.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:34:15.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:34:15.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:34:15.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:34:16.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:34:16.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:34:16.409 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:34:16.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:34:16.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:34:16.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:34:16.691 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 27 up_thru 27 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6802/3631160648,v1:127.0.0.1:6803/3631160648] [v2:127.0.0.1:6804/3631160648,v1:127.0.0.1:6805/3631160648] exists,up 24af3092-66bf-4d8b-a4ed-e30b0b55968f 2024-08-19T15:34:16.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:34:16.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:34:16.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:34:16.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T15:34:16.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:34:16.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:34:16.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:34:16.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:34:16.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:34:16.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:34:16.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:34:16.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:34:16.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:34:16.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:34:16.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:34:16.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:34:16.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:34:16.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:34:16.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:34:16.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:34:16.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:34:16.696 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:34:16.697 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:34:16.697 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:34:16.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:34:16.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:34:16.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:34:16.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:34:16.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:34:16.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:34:16.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:34:16.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:34:16.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:34:16.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:34:16.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-08-19T15:34:16.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:34:16.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:34:16.698 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:34:16.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-08-19T15:34:16.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:34:16.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:34:16.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:34:16.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:34:16.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:34:16.721 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:16.720+0000 7f3756142640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:34:16.723 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:16.722+0000 7f3756142640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:34:16.724 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:16.723+0000 7f3756142640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:34:16.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:34:16.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:34:16.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:34:16.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:34:17.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:34:17.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:34:17.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:34:17.000 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:34:17.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:34:17.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:34:17.248 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:17.247+0000 7f3756142640 -1 Falling back to public interface 2024-08-19T15:34:17.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:34:18.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:34:18.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:34:18.266 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:34:18.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:34:18.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:34:18.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:34:18.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:34:19.547 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:19.546+0000 7f3756142640 -1 osd.0 21 log_to_monitors true 2024-08-19T15:34:19.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:34:19.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:34:19.608 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:34:19.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:34:19.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:34:19.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:34:19.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:34:20.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:34:20.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:34:20.883 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:34:20.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:34:20.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:34:20.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:34:21.172 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6810/2760383662,v1:127.0.0.1:6811/2760383662] [v2:127.0.0.1:6812/2760383662,v1:127.0.0.1:6813/2760383662] exists,up 2d9ab22d-cbc4-46e1-8076-0ea490280584 2024-08-19T15:34:21.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:34:21.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:34:21.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:34:21.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:768: TEST_repair_stats_ec: wait_for_clean 2024-08-19T15:34:21.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:34:21.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:34:21.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:34:21.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:34:21.174 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:34:21.174 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:34:21.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:34:21.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:34:21.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:34:21.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:34:21.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:34:21.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:34:21.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:34:21.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:34:21.269 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:34:21.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:34:21.533 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:34:21.533 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:34:21.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:34:21.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:34:21.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:34:21.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-08-19T15:34:21.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-08-19T15:34:21.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-08-19T15:34:21.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:34:21.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:34:21.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964116995 2024-08-19T15:34:21.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964116995 2024-08-19T15:34:21.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-115964116995' 2024-08-19T15:34:21.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:34:21.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:34:21.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-08-19T15:34:21.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-08-19T15:34:21.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-115964116995 2-64424509457' 2024-08-19T15:34:21.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:34:21.841 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-08-19T15:34:21.841 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:34:21.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:34:21.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-08-19T15:34:21.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:34:21.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-08-19T15:34:21.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-08-19T15:34:21.844 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 128849018882 2024-08-19T15:34:21.845 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:34:22.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-08-19T15:34:22.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:34:23.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:34:23.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:34:23.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-08-19T15:34:23.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:34:23.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-115964116995 2024-08-19T15:34:23.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:34:23.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:34:23.388 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-115964116995 2024-08-19T15:34:23.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:34:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964116995 2024-08-19T15:34:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 115964116995' 2024-08-19T15:34:23.390 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 115964116995 2024-08-19T15:34:23.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:34:23.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964116995 -lt 115964116995 2024-08-19T15:34:23.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:34:23.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-08-19T15:34:23.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:34:23.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:34:23.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-08-19T15:34:23.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:34:23.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-08-19T15:34:23.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-08-19T15:34:23.660 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509457 2024-08-19T15:34:23.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:34:23.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-08-19T15:34:23.928 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:34:23.928 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:34:23.929 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:34:24.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:34:24.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:34:24.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:34:24.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:34:24.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:34:24.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:34:24.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:34:24.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:34:24.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:34:24.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:34:24.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:34:24.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:34:24.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:34:24.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:34:24.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:34:24.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats_ec: repair 1.0 2024-08-19T15:34:24.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-08-19T15:34:24.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-08-19T15:34:24.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:34:24.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:34:24.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:34:24.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:34:25.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T15:33:24.561562+0000 2024-08-19T15:34:25.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-08-19T15:34:25.369 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-08-19T15:34:25.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-08-19T15:33:24.561562+0000 2024-08-19T15:34:25.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-08-19T15:34:25.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:33:24.561562+0000 2024-08-19T15:34:25.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:34:25.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:34:25.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:34:25.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:34:25.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:34:25.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:34:25.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:34:25.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:34:25.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:33:24.561562+0000 '>' 2024-08-19T15:33:24.561562+0000 2024-08-19T15:34:25.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:34:26.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:34:26.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:34:26.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:34:26.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:34:26.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:34:26.648 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:34:26.648 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:34:26.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:33:24.561562+0000 '>' 2024-08-19T15:33:24.561562+0000 2024-08-19T15:34:26.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:34:27.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:34:27.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:34:27.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:34:27.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:34:27.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:34:27.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:34:27.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:34:28.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:33:24.561562+0000 '>' 2024-08-19T15:33:24.561562+0000 2024-08-19T15:34:28.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:34:29.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:34:29.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:34:29.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:34:29.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:34:29.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:34:29.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:34:29.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:34:29.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:33:24.561562+0000 '>' 2024-08-19T15:33:24.561562+0000 2024-08-19T15:34:29.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:34:30.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:34:30.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:34:30.534 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:34:30.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:34:30.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:34:30.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:34:30.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:34:30.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:33:24.561562+0000 '>' 2024-08-19T15:33:24.561562+0000 2024-08-19T15:34:30.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:34:31.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:34:31.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:34:31.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:34:31.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:34:31.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:34:31.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:34:31.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:34:32.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:33:24.561562+0000 '>' 2024-08-19T15:33:24.561562+0000 2024-08-19T15:34:32.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:34:33.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:34:33.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:34:33.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:34:33.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:34:33.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:34:33.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:34:33.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:34:33.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:34:26.114695+0000 '>' 2024-08-19T15:33:24.561562+0000 2024-08-19T15:34:33.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:34:33.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats_ec: wait_for_clean 2024-08-19T15:34:33.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:34:33.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:34:33.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:34:33.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:34:33.358 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:34:33.358 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:34:33.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:34:33.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:34:33.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:34:33.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:34:33.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:34:33.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:34:33.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:34:33.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:34:33.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:34:33.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:34:33.764 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:34:33.764 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:34:33.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:34:33.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:34:33.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:34:33.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018885 2024-08-19T15:34:33.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018885 2024-08-19T15:34:33.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885' 2024-08-19T15:34:33.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:34:33.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:34:33.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964116998 2024-08-19T15:34:33.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964116998 2024-08-19T15:34:33.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-115964116998' 2024-08-19T15:34:33.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:34:33.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:34:34.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-08-19T15:34:34.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-08-19T15:34:34.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-115964116998 2-64424509461' 2024-08-19T15:34:34.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:34:34.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018885 2024-08-19T15:34:34.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:34:34.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:34:34.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018885 2024-08-19T15:34:34.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:34:34.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018885 2024-08-19T15:34:34.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2024-08-19T15:34:34.074 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 128849018885 2024-08-19T15:34:34.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:34:34.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018884 -lt 128849018885 2024-08-19T15:34:34.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:34:35.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:34:35.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:34:35.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018885 -lt 128849018885 2024-08-19T15:34:35.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:34:35.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-115964116998 2024-08-19T15:34:35.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:34:35.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:34:35.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-115964116998 2024-08-19T15:34:35.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:34:35.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964116998 2024-08-19T15:34:35.616 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 115964116998 2024-08-19T15:34:35.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 115964116998' 2024-08-19T15:34:35.617 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:34:35.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964116998 -lt 115964116998 2024-08-19T15:34:35.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:34:35.886 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-08-19T15:34:35.886 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:34:35.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:34:35.887 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-08-19T15:34:35.887 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:34:35.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-08-19T15:34:35.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-08-19T15:34:35.889 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509461 2024-08-19T15:34:35.889 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:34:36.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-08-19T15:34:36.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:34:36.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:34:36.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:34:36.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:34:36.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:34:36.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:34:36.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:34:36.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:34:36.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:34:36.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:34:36.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:34:36.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:34:36.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:34:36.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:34:36.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:34:37.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:34:37.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:34:37.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:34:37.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:772: TEST_repair_stats_ec: ceph pg dump pgs 2024-08-19T15:34:37.346 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T15:34:37.346 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-08-19T15:34:27.698933+0000 21'30 31:135 [1,0,2] 1 [1,0,2] 1 21'30 2024-08-19T15:34:26.114695+0000 21'30 2024-08-19T15:34:26.114695+0000 0 1 periodic scrub scheduled @ 2024-08-20T15:34:26.114695+0000 17 0 2024-08-19T15:34:37.346 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T15:34:37.346 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T15:34:37.346 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:34:37.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats_ec: flush_pg_stats 2024-08-19T15:34:37.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:34:37.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:34:37.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:34:37.625 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:34:37.625 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:34:37.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:34:37.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:34:37.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:34:37.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018887 2024-08-19T15:34:37.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018887 2024-08-19T15:34:37.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018887' 2024-08-19T15:34:37.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:34:37.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:34:37.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117000 2024-08-19T15:34:37.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117000 2024-08-19T15:34:37.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018887 1-115964117000' 2024-08-19T15:34:37.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:34:37.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:34:37.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-08-19T15:34:37.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-08-19T15:34:37.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018887 1-115964117000 2-64424509462' 2024-08-19T15:34:37.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:34:37.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018887 2024-08-19T15:34:37.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:34:37.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:34:37.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018887 2024-08-19T15:34:37.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:34:37.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018887 2024-08-19T15:34:37.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018887' 2024-08-19T15:34:37.963 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 128849018887 2024-08-19T15:34:37.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:34:38.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018886 -lt 128849018887 2024-08-19T15:34:38.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:34:39.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:34:39.227 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:34:39.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018887 2024-08-19T15:34:39.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:34:39.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-115964117000 2024-08-19T15:34:39.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:34:39.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:34:39.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-115964117000 2024-08-19T15:34:39.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:34:39.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117000 2024-08-19T15:34:39.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 115964117000' 2024-08-19T15:34:39.517 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 115964117000 2024-08-19T15:34:39.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:34:39.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117000 -lt 115964117000 2024-08-19T15:34:39.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:34:39.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-08-19T15:34:39.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:34:39.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:34:39.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-08-19T15:34:39.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:34:39.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-08-19T15:34:39.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-08-19T15:34:39.788 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509462 2024-08-19T15:34:39.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:34:40.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509462 2024-08-19T15:34:40.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: ceph pg 1.0 query 2024-08-19T15:34:40.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: jq .info.stats.stat_sum 2024-08-19T15:34:40.208 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:34:40.209 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes": 210, 2024-08-19T15:34:40.209 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects": 30, 2024-08-19T15:34:40.209 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_clones": 0, 2024-08-19T15:34:40.209 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_copies": 90, 2024-08-19T15:34:40.209 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing_on_primary": 0, 2024-08-19T15:34:40.209 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing": 0, 2024-08-19T15:34:40.209 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_degraded": 0, 2024-08-19T15:34:40.209 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_misplaced": 0, 2024-08-19T15:34:40.209 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_unfound": 0, 2024-08-19T15:34:40.209 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_dirty": 30, 2024-08-19T15:34:40.209 INFO:tasks.workunit.client.0.smithi102.stdout: "num_whiteouts": 0, 2024-08-19T15:34:40.210 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read": 0, 2024-08-19T15:34:40.210 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read_kb": 0, 2024-08-19T15:34:40.210 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write": 30, 2024-08-19T15:34:40.210 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write_kb": 30, 2024-08-19T15:34:40.210 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrub_errors": 0, 2024-08-19T15:34:40.210 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shallow_scrub_errors": 0, 2024-08-19T15:34:40.210 INFO:tasks.workunit.client.0.smithi102.stdout: "num_deep_scrub_errors": 0, 2024-08-19T15:34:40.210 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_recovered": 26, 2024-08-19T15:34:40.210 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_recovered": 182, 2024-08-19T15:34:40.210 INFO:tasks.workunit.client.0.smithi102.stdout: "num_keys_recovered": 0, 2024-08-19T15:34:40.210 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_omap": 0, 2024-08-19T15:34:40.210 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_hit_set_archive": 0, 2024-08-19T15:34:40.210 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_hit_set_archive": 0, 2024-08-19T15:34:40.211 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush": 0, 2024-08-19T15:34:40.211 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_kb": 0, 2024-08-19T15:34:40.211 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict": 0, 2024-08-19T15:34:40.211 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_kb": 0, 2024-08-19T15:34:40.211 INFO:tasks.workunit.client.0.smithi102.stdout: "num_promote": 0, 2024-08-19T15:34:40.211 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_high": 0, 2024-08-19T15:34:40.211 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_low": 0, 2024-08-19T15:34:40.211 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_some": 0, 2024-08-19T15:34:40.211 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_full": 0, 2024-08-19T15:34:40.211 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_pinned": 0, 2024-08-19T15:34:40.211 INFO:tasks.workunit.client.0.smithi102.stdout: "num_legacy_snapsets": 0, 2024-08-19T15:34:40.211 INFO:tasks.workunit.client.0.smithi102.stdout: "num_large_omap_objects": 0, 2024-08-19T15:34:40.211 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_manifest": 0, 2024-08-19T15:34:40.212 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_bytes": 0, 2024-08-19T15:34:40.212 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_keys": 0, 2024-08-19T15:34:40.212 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_repaired": 26 2024-08-19T15:34:40.212 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:34:40.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: ceph pg 1.0 query 2024-08-19T15:34:40.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2024-08-19T15:34:40.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: COUNT=26 2024-08-19T15:34:40.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: test 26 = 26 2024-08-19T15:34:40.314 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: expr 3 - 1 2024-08-19T15:34:40.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: seq 0 2 2024-08-19T15:34:40.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:34:40.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2024-08-19T15:34:40.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-08-19T15:34:40.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-08-19T15:34:40.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-08-19T15:34:40.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-08-19T15:34:40.570 INFO:tasks.workunit.client.0.smithi102.stderr:dumped all 2024-08-19T15:34:40.581 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:34:40.581 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:34:40.581 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 30, 2024-08-19T15:34:40.581 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 128849018888, 2024-08-19T15:34:40.581 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 1, 2024-08-19T15:34:40.581 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 1, 2024-08-19T15:34:40.581 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 1, 2024-08-19T15:34:40.581 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 1, 2024-08-19T15:34:40.581 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 104857600, 2024-08-19T15:34:40.581 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 28480, 2024-08-19T15:34:40.581 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 376, 2024-08-19T15:34:40.582 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 4, 2024-08-19T15:34:40.582 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 28091, 2024-08-19T15:34:40.582 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 104829120, 2024-08-19T15:34:40.582 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-08-19T15:34:40.582 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 107374182400, 2024-08-19T15:34:40.582 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 107345018880, 2024-08-19T15:34:40.582 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-08-19T15:34:40.582 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 385024, 2024-08-19T15:34:40.582 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 206078, 2024-08-19T15:34:40.582 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-08-19T15:34:40.582 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-08-19T15:34:40.582 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-08-19T15:34:40.582 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 4402, 2024-08-19T15:34:40.583 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 28765902 2024-08-19T15:34:40.583 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:34:40.583 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [ 2024-08-19T15:34:40.583 INFO:tasks.workunit.client.0.smithi102.stdout: 1, 2024-08-19T15:34:40.583 INFO:tasks.workunit.client.0.smithi102.stdout: 2 2024-08-19T15:34:40.583 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:34:40.583 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-08-19T15:34:40.583 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-08-19T15:34:40.583 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 13, 2024-08-19T15:34:40.583 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-08-19T15:34:40.583 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-08-19T15:34:40.583 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-08-19T15:34:40.583 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:34:40.583 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-08-19T15:34:40.584 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-08-19T15:34:40.584 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-08-19T15:34:40.584 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-08-19T15:34:40.584 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-08-19T15:34:40.584 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:34:40.584 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [] 2024-08-19T15:34:40.584 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:34:40.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-08-19T15:34:40.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-08-19T15:34:40.831 INFO:tasks.workunit.client.0.smithi102.stderr:dumped all 2024-08-19T15:34:40.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=13 2024-08-19T15:34:40.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 13 = 13 2024-08-19T15:34:40.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:34:40.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 1 = 0 -o 1 = 1 ']' 2024-08-19T15:34:40.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-08-19T15:34:40.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-08-19T15:34:40.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-08-19T15:34:40.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-08-19T15:34:41.094 INFO:tasks.workunit.client.0.smithi102.stderr:dumped all 2024-08-19T15:34:41.106 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:34:41.106 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:34:41.106 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 27, 2024-08-19T15:34:41.106 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 115964117000, 2024-08-19T15:34:41.106 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 1, 2024-08-19T15:34:41.107 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 1, 2024-08-19T15:34:41.107 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 1, 2024-08-19T15:34:41.107 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 1, 2024-08-19T15:34:41.107 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 104857600, 2024-08-19T15:34:41.107 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 28480, 2024-08-19T15:34:41.107 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 376, 2024-08-19T15:34:41.107 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 4, 2024-08-19T15:34:41.107 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 28091, 2024-08-19T15:34:41.107 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 104829120, 2024-08-19T15:34:41.107 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-08-19T15:34:41.107 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 107374182400, 2024-08-19T15:34:41.107 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 107345018880, 2024-08-19T15:34:41.107 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-08-19T15:34:41.108 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 385024, 2024-08-19T15:34:41.108 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 206078, 2024-08-19T15:34:41.108 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-08-19T15:34:41.108 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-08-19T15:34:41.108 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-08-19T15:34:41.108 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 4398, 2024-08-19T15:34:41.108 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 28765906 2024-08-19T15:34:41.108 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:34:41.108 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [ 2024-08-19T15:34:41.108 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-08-19T15:34:41.108 INFO:tasks.workunit.client.0.smithi102.stdout: 2 2024-08-19T15:34:41.108 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:34:41.108 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-08-19T15:34:41.108 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-08-19T15:34:41.109 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 13, 2024-08-19T15:34:41.109 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-08-19T15:34:41.109 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-08-19T15:34:41.109 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-08-19T15:34:41.109 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:34:41.109 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-08-19T15:34:41.109 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-08-19T15:34:41.109 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-08-19T15:34:41.109 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-08-19T15:34:41.109 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-08-19T15:34:41.109 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:34:41.109 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [] 2024-08-19T15:34:41.109 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:34:41.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-08-19T15:34:41.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-08-19T15:34:41.355 INFO:tasks.workunit.client.0.smithi102.stderr:dumped all 2024-08-19T15:34:41.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=13 2024-08-19T15:34:41.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 13 = 13 2024-08-19T15:34:41.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:34:41.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 2 = 0 -o 2 = 1 ']' 2024-08-19T15:34:41.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:784: TEST_repair_stats_ec: repair=0 2024-08-19T15:34:41.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-08-19T15:34:41.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 )' 2024-08-19T15:34:41.619 INFO:tasks.workunit.client.0.smithi102.stderr:dumped all 2024-08-19T15:34:41.630 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:34:41.630 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-08-19T15:34:41.630 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 15, 2024-08-19T15:34:41.630 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 64424509463, 2024-08-19T15:34:41.630 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 1, 2024-08-19T15:34:41.630 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 1, 2024-08-19T15:34:41.631 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 1, 2024-08-19T15:34:41.631 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 1, 2024-08-19T15:34:41.631 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 104857600, 2024-08-19T15:34:41.631 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 27208, 2024-08-19T15:34:41.631 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 376, 2024-08-19T15:34:41.631 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 1, 2024-08-19T15:34:41.631 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 26814, 2024-08-19T15:34:41.631 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 104830392, 2024-08-19T15:34:41.631 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-08-19T15:34:41.631 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 107374182400, 2024-08-19T15:34:41.631 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 107346321408, 2024-08-19T15:34:41.631 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-08-19T15:34:41.632 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 385024, 2024-08-19T15:34:41.632 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 206078, 2024-08-19T15:34:41.632 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-08-19T15:34:41.632 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-08-19T15:34:41.632 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-08-19T15:34:41.632 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 1595, 2024-08-19T15:34:41.632 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 27457989 2024-08-19T15:34:41.632 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:34:41.632 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [ 2024-08-19T15:34:41.632 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-08-19T15:34:41.632 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:34:41.632 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:34:41.632 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-08-19T15:34:41.633 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-08-19T15:34:41.633 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 0, 2024-08-19T15:34:41.633 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-08-19T15:34:41.633 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-08-19T15:34:41.633 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-08-19T15:34:41.633 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:34:41.633 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-08-19T15:34:41.633 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-08-19T15:34:41.633 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-08-19T15:34:41.633 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-08-19T15:34:41.633 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-08-19T15:34:41.633 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:34:41.633 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [] 2024-08-19T15:34:41.633 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:34:41.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-08-19T15:34:41.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 ).num_shards_repaired' 2024-08-19T15:34:41.887 INFO:tasks.workunit.client.0.smithi102.stderr:dumped all 2024-08-19T15:34:41.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=0 2024-08-19T15:34:41.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 0 = 0 2024-08-19T15:34:41.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:792: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-08-19T15:34:41.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:792: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum 2024-08-19T15:34:42.146 INFO:tasks.workunit.client.0.smithi102.stderr:dumped all 2024-08-19T15:34:42.156 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:34:42.157 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 0, 2024-08-19T15:34:42.157 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-08-19T15:34:42.157 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 3, 2024-08-19T15:34:42.157 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 3, 2024-08-19T15:34:42.157 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 3, 2024-08-19T15:34:42.157 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 3, 2024-08-19T15:34:42.157 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 314572800, 2024-08-19T15:34:42.157 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 84168, 2024-08-19T15:34:42.157 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 1128, 2024-08-19T15:34:42.158 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 10, 2024-08-19T15:34:42.158 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 82997, 2024-08-19T15:34:42.158 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 314488632, 2024-08-19T15:34:42.158 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-08-19T15:34:42.158 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 322122547200, 2024-08-19T15:34:42.158 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 322036359168, 2024-08-19T15:34:42.158 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-08-19T15:34:42.158 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 1155072, 2024-08-19T15:34:42.158 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 618234, 2024-08-19T15:34:42.159 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-08-19T15:34:42.159 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-08-19T15:34:42.159 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-08-19T15:34:42.159 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 10395, 2024-08-19T15:34:42.159 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 84989797 2024-08-19T15:34:42.159 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:34:42.159 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [], 2024-08-19T15:34:42.159 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-08-19T15:34:42.159 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-08-19T15:34:42.159 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 26, 2024-08-19T15:34:42.160 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-08-19T15:34:42.160 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-08-19T15:34:42.160 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-08-19T15:34:42.160 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:34:42.160 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-08-19T15:34:42.160 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-08-19T15:34:42.160 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-08-19T15:34:42.160 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-08-19T15:34:42.160 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-08-19T15:34:42.160 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:34:42.161 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [], 2024-08-19T15:34:42.161 INFO:tasks.workunit.client.0.smithi102.stdout: "network_ping_times": [] 2024-08-19T15:34:42.161 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:34:42.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-08-19T15:34:42.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-08-19T15:34:42.410 INFO:tasks.workunit.client.0.smithi102.stderr:dumped all 2024-08-19T15:34:42.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: COUNT=26 2024-08-19T15:34:42.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:794: TEST_repair_stats_ec: test 26 = 26 2024-08-19T15:34:42.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T15:34:42.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:34:42.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:34:42.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:34:42.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:34:42.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:34:42.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:34:42.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:34:42.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:34:42.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:34:42.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:34:42.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:34:42.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:34:42.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:34:42.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:34:42.536 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:34:42.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:34:42.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:34:42.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:34:42.536 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:34:42.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:34:42.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:34:42.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:34:42.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:34:42.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:34:42.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:34:42.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:34:42.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:34:42.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:34:42.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T15:34:42.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T15:34:42.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T15:34:42.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T15:34:42.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:34:42.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:34:42.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:34:42.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:34:42.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:34:42.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:34:42.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:34:42.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:34:42.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:34:42.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:34:42.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:34:42.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:34:42.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:34:42.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:34:42.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:34:42.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:34:42.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:34:42.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:34:42.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:34:42.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:34:42.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:34:42.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:34:42.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:34:42.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:34:42.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:34:42.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:34:42.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:34:42.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:34:42.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T15:34:42.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T15:34:42.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:34:42.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:34:42.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T15:34:42.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T15:34:42.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T15:34:42.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T15:34:42.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T15:34:42.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_request_scrub_priority td/osd-scrub-repair 2024-08-19T15:34:42.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6197: TEST_request_scrub_priority: local dir=td/osd-scrub-repair 2024-08-19T15:34:42.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6198: TEST_request_scrub_priority: local poolname=psr_pool 2024-08-19T15:34:42.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6199: TEST_request_scrub_priority: local objname=POBJ 2024-08-19T15:34:42.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6200: TEST_request_scrub_priority: local OBJECTS=64 2024-08-19T15:34:42.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6201: TEST_request_scrub_priority: local PGS=8 2024-08-19T15:34:42.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6203: TEST_request_scrub_priority: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T15:34:42.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T15:34:42.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T15:34:42.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T15:34:42.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T15:34:42.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T15:34:42.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T15:34:42.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T15:34:42.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:34:42.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:34:42.884 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:34:42.884 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:34:42.884 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:34:42.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:34:42.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T15:34:43.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T15:34:43.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T15:34:43.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:34:43.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:34:43.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T15:34:43.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T15:34:43.067 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:34:43.068 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:34:43.068 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:34:43.068 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:34:43.068 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:34:43.068 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:34:43.068 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:34:43.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:34:43.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T15:34:43.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T15:34:43.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:34:43.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:34:43.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T15:34:43.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T15:34:43.149 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:34:43.149 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:34:43.149 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:34:43.150 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:34:43.150 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:34:43.150 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:34:43.150 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:34:43.150 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:34:43.150 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T15:34:43.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6204: TEST_request_scrub_priority: run_mgr td/osd-scrub-repair x 2024-08-19T15:34:43.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T15:34:43.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T15:34:43.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T15:34:43.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T15:34:43.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T15:34:43.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T15:34:44.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T15:34:44.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:34:44.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:34:44.034 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:34:44.034 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:34:44.034 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:34:44.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:34:44.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:34:44.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:34:44.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6205: TEST_request_scrub_priority: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-08-19T15:34:44.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6206: TEST_request_scrub_priority: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-08-19T15:34:44.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6207: TEST_request_scrub_priority: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-08-19T15:34:44.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:34:44.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:34:44.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T15:34:44.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:34:44.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:34:44.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:34:44.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:34:44.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:34:44.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:34:44.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:34:44.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:34:44.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:34:44.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:34:44.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:34:44.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:34:44.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:34:44.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:34:44.067 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:34:44.067 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:34:44.067 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:34:44.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:34:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:34:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:34:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:34:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:34:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:34:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:34:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:34:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:34:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:34:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:34:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:34:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-08-19T15:34:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:34:44.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:34:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=358de7db-1f17-4303-9914-0a2b5410c673 2024-08-19T15:34:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 358de7db-1f17-4303-9914-0a2b5410c673' 2024-08-19T15:34:44.069 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 358de7db-1f17-4303-9914-0a2b5410c673 2024-08-19T15:34:44.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:34:44.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAUZsNmnTQHBRAAj1npPkXk24cegj53hlVkGQ== 2024-08-19T15:34:44.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAUZsNmnTQHBRAAj1npPkXk24cegj53hlVkGQ=="}' 2024-08-19T15:34:44.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 358de7db-1f17-4303-9914-0a2b5410c673 -i td/osd-scrub-repair/0/new.json 2024-08-19T15:34:44.282 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:34:44.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T15:34:44.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQAUZsNmnTQHBRAAj1npPkXk24cegj53hlVkGQ== --osd-uuid 358de7db-1f17-4303-9914-0a2b5410c673 2024-08-19T15:34:44.323 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:44.322+0000 7f56f6810640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:34:44.332 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:44.332+0000 7f56f6810640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:34:44.334 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:44.333+0000 7f56f6810640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:34:44.334 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:44.334+0000 7f56f6810640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:34:44.335 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:44.334+0000 7f56f6810640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:34:44.335 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:44.334+0000 7f56f6810640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:34:44.424 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:44.423+0000 7f56f6810640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T15:34:49.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T15:34:49.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:34:49.952 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T15:34:49.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T15:34:49.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:34:50.300 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:34:50.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T15:34:50.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-08-19T15:34:50.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:34:50.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:34:50.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:34:50.330 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:50.329+0000 7fa47e616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:34:50.332 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:50.331+0000 7fa47e616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:34:50.333 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:50.332+0000 7fa47e616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:34:50.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T15:34:50.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:34:50.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:34:50.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:34:50.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:34:50.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:34:50.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:34:50.573 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:34:50.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:34:50.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:34:50.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:34:50.856 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:50.855+0000 7fa47e616640 -1 Falling back to public interface 2024-08-19T15:34:51.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:34:51.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:34:51.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:34:51.841 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:34:51.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:34:51.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:34:52.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:34:53.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:34:53.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:34:53.115 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:34:53.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:34:53.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:34:53.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:34:53.167 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:34:53.165+0000 7fa47e616640 -1 osd.0 0 log_to_monitors true 2024-08-19T15:34:53.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:34:54.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:34:54.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:34:54.427 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:34:54.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:34:54.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:34:54.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:34:54.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:34:55.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:34:55.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:34:55.728 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:34:55.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:34:55.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:34:55.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:34:56.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:34:57.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:34:57.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:34:57.114 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:34:57.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:34:57.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:34:57.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:34:57.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:34:58.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:34:58.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:34:58.387 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:34:58.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:34:58.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:34:58.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:34:58.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:34:59.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:34:59.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:34:59.659 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:34:59.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:34:59.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:34:59.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:34:59.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:35:00.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:35:00.937 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:35:00.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:35:00.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:35:00.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:35:00.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:35:01.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:35:02.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:35:02.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:35:02.216 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:35:02.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:35:02.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:35:02.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:35:02.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:35:03.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:35:03.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:35:03.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:35:03.494 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:35:03.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:35:03.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:35:03.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:35:04.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:35:04.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:35:04.769 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:35:04.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:35:04.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:35:04.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:35:05.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:35:06.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:35:06.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:35:06.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:35:06.094 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:35:06.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:35:06.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:35:06.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:35:07.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:35:07.388 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:35:07.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:35:07.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:35:07.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:35:07.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:35:07.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:35:08.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:35:08.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:35:08.833 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:35:08.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:35:08.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:35:08.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:35:09.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:35:10.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:35:10.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:35:10.629 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:35:10.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:35:10.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:35:10.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:35:11.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:35:12.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:35:12.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:35:12.505 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:35:12.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:35:12.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:35:12.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:35:13.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:35:14.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:35:14.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:35:14.141 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:35:14.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:35:14.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:35:14.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:35:14.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:35:15.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:35:15.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:35:15.610 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:35:15.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:35:15.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:35:15.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:35:15.881 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4016877558,v1:127.0.0.1:6803/4016877558] [v2:127.0.0.1:6804/4016877558,v1:127.0.0.1:6805/4016877558] exists,up 358de7db-1f17-4303-9914-0a2b5410c673 2024-08-19T15:35:15.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:35:15.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:35:15.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:35:15.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6209: TEST_request_scrub_priority: create_pool psr_pool 8 8 2024-08-19T15:35:15.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool 8 8 2024-08-19T15:35:16.371 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'psr_pool' created 2024-08-19T15:35:16.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:35:17.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6210: TEST_request_scrub_priority: wait_for_clean 2024-08-19T15:35:17.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:35:17.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:35:17.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:35:17.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:35:17.386 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:35:17.386 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:35:17.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:35:17.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:35:17.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:35:17.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:35:17.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:35:17.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:35:17.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:35:17.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:35:17.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:35:17.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-08-19T15:35:17.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:35:17.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:35:17.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:35:17.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-08-19T15:35:17.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-08-19T15:35:17.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-08-19T15:35:17.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:35:17.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-08-19T15:35:17.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:35:17.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:35:17.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-08-19T15:35:17.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:35:17.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-08-19T15:35:17.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-08-19T15:35:17.868 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836489 2024-08-19T15:35:17.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:35:18.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836489 2024-08-19T15:35:18.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:35:18.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:35:18.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:35:18.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-08-19T15:35:18.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:35:18.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:35:18.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:35:18.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:35:18.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:35:18.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:35:18.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:35:19.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-08-19T15:35:19.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:35:19.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:35:19.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:35:19.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-08-19T15:35:19.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:35:19.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:35:19.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6212: TEST_request_scrub_priority: local osd=0 2024-08-19T15:35:19.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6213: TEST_request_scrub_priority: add_something td/osd-scrub-repair psr_pool POBJ noscrub 2024-08-19T15:35:19.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:35:19.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2024-08-19T15:35:19.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2024-08-19T15:35:19.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:35:19.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:35:19.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:35:19.731 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:35:19.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:35:20.158 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:35:20.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:35:20.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:35:20.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-08-19T15:35:20.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6214: TEST_request_scrub_priority: get_primary psr_pool POBJ 2024-08-19T15:35:20.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=psr_pool 2024-08-19T15:35:20.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=POBJ 2024-08-19T15:35:20.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map psr_pool POBJ 2024-08-19T15:35:20.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:35:20.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6214: TEST_request_scrub_priority: local primary=0 2024-08-19T15:35:20.595 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6215: TEST_request_scrub_priority: get_pg psr_pool POBJ 2024-08-19T15:35:20.595 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=psr_pool 2024-08-19T15:35:20.595 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=POBJ 2024-08-19T15:35:20.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map psr_pool POBJ 2024-08-19T15:35:20.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:35:20.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6215: TEST_request_scrub_priority: local pg=1.1 2024-08-19T15:35:20.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6216: TEST_request_scrub_priority: ceph osd dump 2024-08-19T15:35:20.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6216: TEST_request_scrub_priority: grep '^pool.*['\'']psr_pool['\'']' 2024-08-19T15:35:20.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6216: TEST_request_scrub_priority: awk '{ print $2 }' 2024-08-19T15:35:21.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6216: TEST_request_scrub_priority: poolid=1 2024-08-19T15:35:21.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6218: TEST_request_scrub_priority: local otherpgs 2024-08-19T15:35:21.130 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: expr 8 - 1 2024-08-19T15:35:21.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: seq 0 7 2024-08-19T15:35:21.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-08-19T15:35:21.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.0 2024-08-19T15:35:21.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.0 = 1.1 ']' 2024-08-19T15:35:21.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 ' 2024-08-19T15:35:21.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-08-19T15:35:21.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-08-19T15:35:21.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:21.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:21.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-08-19T15:35:21.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:21.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.0 schedule-scrub 1.0 2024-08-19T15:35:21.495 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:35:21.496 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": false, 2024-08-19T15:35:21.496 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:35:21.496 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-12T15:33:41.495490+0000" 2024-08-19T15:35:21.496 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:35:21.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-08-19T15:35:21.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.1 2024-08-19T15:35:21.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.1 = 1.1 ']' 2024-08-19T15:35:21.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6223: TEST_request_scrub_priority: continue 2024-08-19T15:35:21.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-08-19T15:35:21.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.2 2024-08-19T15:35:21.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.2 = 1.1 ']' 2024-08-19T15:35:21.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 1.2 ' 2024-08-19T15:35:21.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-08-19T15:35:21.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-08-19T15:35:21.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:21.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:21.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-08-19T15:35:21.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:21.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.2 schedule-scrub 1.2 2024-08-19T15:35:21.864 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:35:21.865 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": false, 2024-08-19T15:35:21.865 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:35:21.865 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-12T15:33:41.864561+0000" 2024-08-19T15:35:21.865 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:35:21.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-08-19T15:35:21.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.3 2024-08-19T15:35:21.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.3 = 1.1 ']' 2024-08-19T15:35:21.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 ' 2024-08-19T15:35:21.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-08-19T15:35:21.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-08-19T15:35:21.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:21.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:21.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-08-19T15:35:22.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:22.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.3 schedule-scrub 1.3 2024-08-19T15:35:22.251 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:35:22.251 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": false, 2024-08-19T15:35:22.251 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:35:22.251 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-12T15:33:42.250790+0000" 2024-08-19T15:35:22.251 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:35:22.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-08-19T15:35:22.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.4 2024-08-19T15:35:22.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.4 = 1.1 ']' 2024-08-19T15:35:22.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 ' 2024-08-19T15:35:22.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-08-19T15:35:22.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-08-19T15:35:22.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:22.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:22.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-08-19T15:35:22.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:22.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.4 schedule-scrub 1.4 2024-08-19T15:35:22.619 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:35:22.620 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": false, 2024-08-19T15:35:22.620 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:35:22.620 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-12T15:33:42.619496+0000" 2024-08-19T15:35:22.620 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:35:22.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-08-19T15:35:22.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.5 2024-08-19T15:35:22.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.5 = 1.1 ']' 2024-08-19T15:35:22.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 ' 2024-08-19T15:35:22.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-08-19T15:35:22.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-08-19T15:35:22.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:22.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:22.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-08-19T15:35:22.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:22.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.5 schedule-scrub 1.5 2024-08-19T15:35:22.989 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:35:22.989 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": false, 2024-08-19T15:35:22.989 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:35:22.989 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-12T15:33:42.988872+0000" 2024-08-19T15:35:22.989 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:35:22.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-08-19T15:35:22.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.6 2024-08-19T15:35:22.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.6 = 1.1 ']' 2024-08-19T15:35:22.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 ' 2024-08-19T15:35:22.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-08-19T15:35:22.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-08-19T15:35:22.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:22.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:22.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-08-19T15:35:23.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:23.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.6 schedule-scrub 1.6 2024-08-19T15:35:23.358 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:35:23.358 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": false, 2024-08-19T15:35:23.358 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:35:23.358 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-12T15:33:43.358153+0000" 2024-08-19T15:35:23.358 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:35:23.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-08-19T15:35:23.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.7 2024-08-19T15:35:23.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.7 = 1.1 ']' 2024-08-19T15:35:23.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 1.7 ' 2024-08-19T15:35:23.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-08-19T15:35:23.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-08-19T15:35:23.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:23.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:23.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-08-19T15:35:23.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:23.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.7 schedule-scrub 1.7 2024-08-19T15:35:23.726 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:35:23.726 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": false, 2024-08-19T15:35:23.726 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:35:23.727 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-12T15:33:43.725767+0000" 2024-08-19T15:35:23.727 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:35:23.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: sleep 15 2024-08-19T15:35:38.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6232: TEST_request_scrub_priority: flush_pg_stats 2024-08-19T15:35:38.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:35:38.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:35:39.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-08-19T15:35:39.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:35:39.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:35:39.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:35:39.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836494 2024-08-19T15:35:39.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836494 2024-08-19T15:35:39.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494' 2024-08-19T15:35:39.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:35:39.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836494 2024-08-19T15:35:39.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:35:39.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:35:39.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836494 2024-08-19T15:35:39.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:35:39.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836494 2024-08-19T15:35:39.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-08-19T15:35:39.157 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836494 2024-08-19T15:35:39.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:35:39.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836494 2024-08-19T15:35:39.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:35:40.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:35:40.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:35:40.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836494 2024-08-19T15:35:40.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6235: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-08-19T15:35:40.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-08-19T15:35:40.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:40.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:40.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-08-19T15:35:40.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6235: TEST_request_scrub_priority: local last_scrub=2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:40.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6236: TEST_request_scrub_priority: ceph tell 1.1 scrub 2024-08-19T15:35:41.057 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:35:41.058 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": false, 2024-08-19T15:35:41.058 INFO:tasks.workunit.client.0.smithi102.stdout: "must": true, 2024-08-19T15:35:41.058 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "0.000000" 2024-08-19T15:35:41.058 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:35:41.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: ceph osd unset noscrub 2024-08-19T15:35:41.423 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is unset 2024-08-19T15:35:41.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6239: TEST_request_scrub_priority: ceph osd unset nodeep-scrub 2024-08-19T15:35:41.925 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is unset 2024-08-19T15:35:41.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6241: TEST_request_scrub_priority: wait_for_scrub 1.1 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:41.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.1 2024-08-19T15:35:41.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:41.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:35:41.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:35:41.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:35:41.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-08-19T15:35:41.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-08-19T15:35:41.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:41.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:41.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-08-19T15:35:42.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:35:16.253325+0000 '>' 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:42.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:35:43.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:35:43.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:35:43.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-08-19T15:35:43.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-08-19T15:35:43.207 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:43.207 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:43.207 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-08-19T15:35:43.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:35:16.253325+0000 '>' 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:43.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:35:44.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:35:44.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:35:44.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-08-19T15:35:44.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-08-19T15:35:44.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:44.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:44.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-08-19T15:35:44.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:35:16.253325+0000 '>' 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:44.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:35:45.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:35:45.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:35:45.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-08-19T15:35:45.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-08-19T15:35:45.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:45.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:45.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-08-19T15:35:46.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:35:41.506559+0000 '>' 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:46.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:35:46.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-08-19T15:35:46.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.0 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:46.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-08-19T15:35:46.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:46.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:35:46.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:35:46.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:35:46.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:35:46.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:35:46.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:46.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:46.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:35:46.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:35:44.393193+0000 '>' 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:46.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:35:46.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-08-19T15:35:46.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.2 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:46.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.2 2024-08-19T15:35:46.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:46.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:35:46.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:35:46.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:35:46.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-08-19T15:35:46.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-08-19T15:35:46.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:46.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:46.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-08-19T15:35:46.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:35:45.384485+0000 '>' 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:46.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:35:46.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-08-19T15:35:46.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.3 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:46.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-08-19T15:35:46.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:46.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:35:46.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:35:46.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:35:46.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T15:35:46.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:35:46.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:46.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:46.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:35:46.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-12T15:33:42.250790+0000 '>' 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:46.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:35:47.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:35:47.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:35:47.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T15:35:47.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:35:47.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:47.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:47.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:35:48.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-12T15:33:42.250790+0000 '>' 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:48.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:35:49.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:35:49.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:35:49.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T15:35:49.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:35:49.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:49.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:49.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:35:49.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-12T15:33:42.250790+0000 '>' 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:49.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:35:50.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:35:50.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:35:50.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T15:35:50.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:35:50.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:50.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:50.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:35:50.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-12T15:33:42.250790+0000 '>' 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:50.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:35:51.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:35:51.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:35:51.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-08-19T15:35:51.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:35:51.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:51.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:51.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:35:51.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:35:46.360364+0000 '>' 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:51.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:35:51.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-08-19T15:35:51.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.4 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:51.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.4 2024-08-19T15:35:51.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:51.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:35:51.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:35:51.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:35:51.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-08-19T15:35:51.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-08-19T15:35:51.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:51.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:51.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-08-19T15:35:52.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:35:47.360189+0000 '>' 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:52.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:35:52.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-08-19T15:35:52.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.5 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:52.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.5 2024-08-19T15:35:52.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:52.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:35:52.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:35:52.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:35:52.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-08-19T15:35:52.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-08-19T15:35:52.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:52.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:52.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-08-19T15:35:52.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:35:48.396767+0000 '>' 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:52.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:35:52.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-08-19T15:35:52.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.6 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:52.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.6 2024-08-19T15:35:52.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:52.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:35:52.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:35:52.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:35:52.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-08-19T15:35:52.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-08-19T15:35:52.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:52.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:52.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-08-19T15:35:52.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:35:49.411415+0000 '>' 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:52.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:35:52.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-08-19T15:35:52.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.7 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:52.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.7 2024-08-19T15:35:52.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:52.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:35:52.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:35:52.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:35:52.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-08-19T15:35:52.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-08-19T15:35:52.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:52.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:52.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-08-19T15:35:53.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:35:50.442126+0000 '>' 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:53.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:35:53.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-08-19T15:35:53.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.1 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:53.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.1 2024-08-19T15:35:53.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:53.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:35:53.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:35:53.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:35:53.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-08-19T15:35:53.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-08-19T15:35:53.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:35:53.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:35:53.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-08-19T15:35:53.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:35:41.506559+0000 '>' 2024-08-19T15:35:16.253325+0000 2024-08-19T15:35:53.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:35:53.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: grep 'log_channel.*scrub ok' td/osd-scrub-repair/osd.0.log 2024-08-19T15:35:53.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: grep -v purged_snaps 2024-08-19T15:35:53.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: head -1 2024-08-19T15:35:53.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: sed 's/.*[[]DBG[]]//' 2024-08-19T15:35:53.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: grep -q 1.1 2024-08-19T15:35:53.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T15:35:53.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:35:53.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:35:53.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:35:53.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:35:53.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:35:53.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:35:53.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:35:53.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:35:53.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:35:53.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:35:53.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:35:53.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:35:53.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:35:53.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:35:53.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:35:53.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:35:53.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:35:53.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:35:53.396 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:35:53.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:35:53.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:35:53.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:35:53.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:35:53.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:35:53.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:35:53.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:35:53.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:35:53.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:35:53.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T15:35:53.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T15:35:53.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T15:35:53.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T15:35:53.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:35:53.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:35:53.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:35:53.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:35:53.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:35:53.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:35:53.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:35:53.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:35:53.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:35:53.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:35:53.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:35:53.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:35:53.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:35:53.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:35:53.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:35:53.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:35:53.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:35:53.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:35:53.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:35:53.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:35:53.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:35:53.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:35:53.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:35:53.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:35:53.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:35:53.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:35:53.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:35:53.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:35:53.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T15:35:53.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T15:35:53.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:35:53.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:35:53.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T15:35:53.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T15:35:53.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T15:35:53.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T15:35:53.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T15:35:53.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_scrub_warning td/osd-scrub-repair 2024-08-19T15:35:53.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5842: TEST_scrub_warning: local dir=td/osd-scrub-repair 2024-08-19T15:35:53.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5843: TEST_scrub_warning: local poolname=psr_pool 2024-08-19T15:35:53.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5844: TEST_scrub_warning: local objname=POBJ 2024-08-19T15:35:53.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5845: TEST_scrub_warning: local scrubs=5 2024-08-19T15:35:53.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5846: TEST_scrub_warning: local deep_scrubs=5 2024-08-19T15:35:53.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5847: TEST_scrub_warning: local i1_day=86400 2024-08-19T15:35:53.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5848: TEST_scrub_warning: calc 86400 '*' 7 2024-08-19T15:35:53.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: calc: awk 'BEGIN{print 86400 * 7}' 2024-08-19T15:35:53.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5848: TEST_scrub_warning: local i7_days=604800 2024-08-19T15:35:53.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5849: TEST_scrub_warning: calc 86400 '*' 14 2024-08-19T15:35:53.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: calc: awk 'BEGIN{print 86400 * 14}' 2024-08-19T15:35:53.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5849: TEST_scrub_warning: local i14_days=1209600 2024-08-19T15:35:53.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5850: TEST_scrub_warning: local overdue=0.5 2024-08-19T15:35:53.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5851: TEST_scrub_warning: calc 604800 + 86400 + '(' 604800 '*' 0.5 ')' 2024-08-19T15:35:53.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: calc: awk 'BEGIN{print 604800 + 86400 + ( 604800 * 0.5 )}' 2024-08-19T15:35:53.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5851: TEST_scrub_warning: local conf_overdue_seconds=993600 2024-08-19T15:35:53.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5852: TEST_scrub_warning: calc 1209600 + 86400 + '(' 1209600 '*' 0.5 ')' 2024-08-19T15:35:53.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: calc: awk 'BEGIN{print 1209600 + 86400 + ( 1209600 * 0.5 )}' 2024-08-19T15:35:53.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5852: TEST_scrub_warning: local pool_overdue_seconds=1900800 2024-08-19T15:35:53.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5854: TEST_scrub_warning: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T15:35:53.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T15:35:53.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T15:35:53.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T15:35:53.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T15:35:53.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T15:35:53.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T15:35:53.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T15:35:53.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:35:53.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:35:53.800 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:35:53.800 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:35:53.800 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:35:53.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:35:53.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T15:35:53.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T15:35:53.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T15:35:53.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:35:53.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:35:53.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T15:35:53.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T15:35:53.995 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:35:53.995 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:35:53.995 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:35:53.995 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:35:53.995 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:35:53.995 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:35:53.995 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:35:53.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:35:53.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T15:35:54.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T15:35:54.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:35:54.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:35:54.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T15:35:54.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T15:35:54.075 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:35:54.075 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:35:54.075 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:35:54.076 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:35:54.076 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:35:54.076 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:35:54.076 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:35:54.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:35:54.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T15:35:54.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5855: TEST_scrub_warning: run_mgr td/osd-scrub-repair x --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2024-08-19T15:35:54.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T15:35:54.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T15:35:54.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T15:35:54.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T15:35:54.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T15:35:54.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T15:35:54.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T15:35:54.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:35:54.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:35:54.961 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:35:54.961 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:35:54.961 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:35:54.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:35:54.962 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:35:54.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2024-08-19T15:35:54.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5856: TEST_scrub_warning: run_osd td/osd-scrub-repair 0 --osd_scrub_backoff_ratio=0 2024-08-19T15:35:54.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:35:54.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:35:54.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T15:35:54.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:35:54.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:35:54.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:35:54.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:35:54.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:35:54.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:35:54.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:35:54.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:35:54.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:35:54.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:35:54.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:35:54.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:35:54.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:35:54.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:35:54.995 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:35:54.995 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:35:54.996 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:35:54.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:35:54.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:35:54.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:35:54.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:35:54.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:35:54.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:35:54.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:35:54.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:35:54.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:35:54.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:35:54.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:35:54.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:35:54.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_backoff_ratio=0 2024-08-19T15:35:54.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:35:54.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:35:54.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d96b9055-6bd8-45b7-bc6e-740e902592e1 2024-08-19T15:35:54.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d96b9055-6bd8-45b7-bc6e-740e902592e1' 2024-08-19T15:35:54.997 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 d96b9055-6bd8-45b7-bc6e-740e902592e1 2024-08-19T15:35:54.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:35:55.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBbZsNmp2/CABAAQSghu6m36qXnaHKG2Z52JQ== 2024-08-19T15:35:55.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBbZsNmp2/CABAAQSghu6m36qXnaHKG2Z52JQ=="}' 2024-08-19T15:35:55.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d96b9055-6bd8-45b7-bc6e-740e902592e1 -i td/osd-scrub-repair/0/new.json 2024-08-19T15:35:55.217 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:35:55.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T15:35:55.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 --mkfs --key AQBbZsNmp2/CABAAQSghu6m36qXnaHKG2Z52JQ== --osd-uuid d96b9055-6bd8-45b7-bc6e-740e902592e1 2024-08-19T15:35:55.257 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:35:55.256+0000 7f5856418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:35:55.268 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:35:55.267+0000 7f5856418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:35:55.269 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:35:55.269+0000 7f5856418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:35:55.270 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:35:55.269+0000 7f5856418640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:35:55.270 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:35:55.269+0000 7f5856418640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:35:55.270 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:35:55.269+0000 7f5856418640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:35:55.360 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:35:55.359+0000 7f5856418640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T15:36:00.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T15:36:00.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:36:00.879 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T15:36:00.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T15:36:00.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:36:01.225 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:36:01.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T15:36:01.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 2024-08-19T15:36:01.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:36:01.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:36:01.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:36:01.253 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:36:01.252+0000 7f4b780fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:36:01.255 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:36:01.254+0000 7f4b780fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:36:01.256 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:36:01.255+0000 7f4b780fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:36:01.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T15:36:01.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:36:01.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:36:01.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:36:01.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:36:01.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:36:01.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:36:01.493 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:36:01.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:36:01.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:36:01.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:36:01.786 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:36:01.785+0000 7f4b780fd640 -1 Falling back to public interface 2024-08-19T15:36:02.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:36:02.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:36:02.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:36:02.760 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:36:02.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:36:02.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:36:03.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:36:04.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:36:04.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:36:04.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:36:04.029 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:36:04.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:36:04.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:36:04.133 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:36:04.132+0000 7f4b780fd640 -1 osd.0 0 log_to_monitors true 2024-08-19T15:36:04.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:36:05.351 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:36:05.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:36:05.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:36:05.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:36:05.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:36:05.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:36:05.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:36:06.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:36:06.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:36:06.639 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:36:06.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:36:06.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:36:06.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:36:06.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:36:07.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:36:07.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:36:07.990 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:36:07.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:36:07.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:36:07.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:36:08.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:36:09.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:36:09.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:36:09.266 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:36:09.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:36:09.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:36:09.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:36:09.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:36:10.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:36:10.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:36:10.542 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:36:10.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:36:10.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:36:10.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:36:10.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:36:11.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:36:11.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:36:11.855 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:36:11.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:36:11.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:36:11.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:36:12.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:36:13.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:36:13.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:36:13.132 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:36:13.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:36:13.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:36:13.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:36:13.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:36:14.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:36:14.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:36:14.410 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:36:14.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:36:14.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:36:14.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:36:14.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:36:15.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:36:15.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:36:15.689 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:36:15.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:36:15.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:36:15.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:36:16.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:36:17.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:36:17.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:36:17.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:36:17.343 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:36:17.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:36:17.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:36:17.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:36:18.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:36:18.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:36:18.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:36:18.846 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:36:18.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:36:18.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:36:19.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:36:20.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:36:20.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:36:20.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:36:20.651 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:36:20.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:36:20.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:36:21.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:36:22.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:36:22.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:36:22.114 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:36:22.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:36:22.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:36:22.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:36:22.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:36:23.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:36:23.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:36:23.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:36:23.874 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:36:23.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:36:23.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:36:24.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:36:25.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:36:25.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:36:25.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:36:25.643 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:36:25.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:36:25.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:36:26.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:36:27.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:36:27.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:36:27.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:36:27.057 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:36:27.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:36:27.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:36:27.347 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3793420559,v1:127.0.0.1:6803/3793420559] [v2:127.0.0.1:6804/3793420559,v1:127.0.0.1:6805/3793420559] exists,up d96b9055-6bd8-45b7-bc6e-740e902592e1 2024-08-19T15:36:27.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:36:27.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:36:27.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:36:27.348 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: expr 5 + 5 2024-08-19T15:36:27.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: seq 1 10 2024-08-19T15:36:27.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-08-19T15:36:27.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-1 1 1 2024-08-19T15:36:27.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-1 1 1 2024-08-19T15:36:27.721 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'psr_pool-1' created 2024-08-19T15:36:27.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:36:28.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-08-19T15:36:28.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:36:28.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:36:28.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:36:28.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:36:28.736 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:36:28.736 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:36:28.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:36:28.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:36:28.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:36:28.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:36:28.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:36:28.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:36:28.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:36:28.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:36:28.841 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:36:29.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-08-19T15:36:29.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:36:29.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:36:29.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:36:29.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836482 2024-08-19T15:36:29.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836482 2024-08-19T15:36:29.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836482' 2024-08-19T15:36:29.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:36:29.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836482 2024-08-19T15:36:29.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:36:29.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:36:29.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836482 2024-08-19T15:36:29.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:36:29.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836482 2024-08-19T15:36:29.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836482' 2024-08-19T15:36:29.367 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836482 2024-08-19T15:36:29.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:36:29.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836481 -lt 21474836482 2024-08-19T15:36:29.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:36:30.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:36:30.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:36:30.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836481 -lt 21474836482 2024-08-19T15:36:30.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:36:31.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:36:31.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:36:32.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836482 -lt 21474836482 2024-08-19T15:36:32.184 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:36:32.185 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:36:32.185 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:36:32.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:36:32.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:36:32.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:36:32.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:36:32.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:36:32.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:36:32.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:36:32.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:36:32.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:36:32.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:36:32.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:36:32.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:36:33.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:36:33.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:36:33.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:36:33.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 1 = 1 ']' 2024-08-19T15:36:33.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: ceph osd pool set psr_pool-1 scrub_max_interval 1209600 2024-08-19T15:36:33.515 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 scrub_max_interval to 1209600 2024-08-19T15:36:33.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-08-19T15:36:33.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 1 = 6 ']' 2024-08-19T15:36:33.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-08-19T15:36:33.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-2 1 1 2024-08-19T15:36:33.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-2 1 1 2024-08-19T15:36:33.892 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'psr_pool-2' created 2024-08-19T15:36:33.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:36:34.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-08-19T15:36:34.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:36:34.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:36:34.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:36:34.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:36:34.906 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:36:34.906 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:36:34.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:36:34.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:36:34.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:36:35.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:36:35.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:36:35.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:36:35.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:36:35.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:36:35.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:36:35.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-08-19T15:36:35.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:36:35.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:36:35.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:36:35.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836484 2024-08-19T15:36:35.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836484 2024-08-19T15:36:35.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836484' 2024-08-19T15:36:35.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:36:35.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836484 2024-08-19T15:36:35.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:36:35.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:36:35.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836484 2024-08-19T15:36:35.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:36:35.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836484 2024-08-19T15:36:35.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-08-19T15:36:35.413 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836484 2024-08-19T15:36:35.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:36:35.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836483 -lt 21474836484 2024-08-19T15:36:35.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:36:36.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:36:36.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:36:36.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836483 -lt 21474836484 2024-08-19T15:36:36.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:36:37.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:36:37.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:36:38.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836484 2024-08-19T15:36:38.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:36:38.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:36:38.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:36:38.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-08-19T15:36:38.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:36:38.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:36:38.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:36:38.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:36:38.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:36:38.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:36:38.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:36:38.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-08-19T15:36:38.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:36:38.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:36:38.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:36:39.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-08-19T15:36:39.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:36:39.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:36:39.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 2 = 1 ']' 2024-08-19T15:36:39.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-08-19T15:36:39.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 2 = 6 ']' 2024-08-19T15:36:39.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-08-19T15:36:39.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-3 1 1 2024-08-19T15:36:39.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-3 1 1 2024-08-19T15:36:39.544 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'psr_pool-3' created 2024-08-19T15:36:39.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:36:40.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-08-19T15:36:40.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:36:40.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:36:40.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:36:40.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:36:40.559 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:36:40.559 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:36:40.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:36:40.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:36:40.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:36:40.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:36:40.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:36:40.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:36:40.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:36:40.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:36:40.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:36:40.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-08-19T15:36:40.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:36:40.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:36:40.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:36:41.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836486 2024-08-19T15:36:41.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836486 2024-08-19T15:36:41.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486' 2024-08-19T15:36:41.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:36:41.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836486 2024-08-19T15:36:41.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:36:41.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:36:41.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836486 2024-08-19T15:36:41.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:36:41.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836486 2024-08-19T15:36:41.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-08-19T15:36:41.057 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836486 2024-08-19T15:36:41.058 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:36:41.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836486 2024-08-19T15:36:41.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:36:42.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:36:42.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:36:42.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836486 2024-08-19T15:36:42.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:36:43.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:36:43.606 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:36:43.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836486 2024-08-19T15:36:43.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:36:43.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:36:43.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:36:44.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-08-19T15:36:44.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:36:44.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:36:44.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:36:44.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:36:44.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:36:44.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:36:44.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:36:44.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-08-19T15:36:44.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:36:44.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:36:44.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:36:44.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-08-19T15:36:44.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:36:44.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:36:44.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 3 = 1 ']' 2024-08-19T15:36:44.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-08-19T15:36:44.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 3 = 6 ']' 2024-08-19T15:36:44.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-08-19T15:36:44.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-4 1 1 2024-08-19T15:36:44.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-4 1 1 2024-08-19T15:36:45.322 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'psr_pool-4' created 2024-08-19T15:36:45.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:36:46.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-08-19T15:36:46.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:36:46.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:36:46.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:36:46.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:36:46.336 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:36:46.336 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:36:46.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:36:46.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:36:46.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:36:46.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:36:46.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:36:46.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:36:46.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:36:46.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:36:46.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:36:46.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-08-19T15:36:46.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:36:46.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:36:46.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:36:46.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-08-19T15:36:46.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-08-19T15:36:46.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-08-19T15:36:46.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:36:46.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-08-19T15:36:46.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:36:46.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:36:46.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-08-19T15:36:46.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:36:46.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-08-19T15:36:46.813 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836489 2024-08-19T15:36:46.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-08-19T15:36:46.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:36:47.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836489 2024-08-19T15:36:47.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:36:47.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:36:47.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:36:47.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T15:36:47.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:36:47.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:36:47.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:36:47.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:36:47.456 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:36:47.456 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:36:47.456 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:36:47.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:36:47.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:36:47.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:36:47.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:36:48.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T15:36:48.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:36:48.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:36:48.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 4 = 1 ']' 2024-08-19T15:36:48.047 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-08-19T15:36:48.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 4 = 6 ']' 2024-08-19T15:36:48.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-08-19T15:36:48.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-5 1 1 2024-08-19T15:36:48.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-5 1 1 2024-08-19T15:36:48.422 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'psr_pool-5' created 2024-08-19T15:36:48.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:36:49.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-08-19T15:36:49.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:36:49.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:36:49.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:36:49.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:36:49.436 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:36:49.436 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:36:49.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:36:49.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:36:49.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:36:49.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:36:49.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:36:49.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:36:49.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:36:49.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:36:49.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:36:49.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-08-19T15:36:49.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:36:49.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:36:49.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:36:49.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836490 2024-08-19T15:36:49.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836490 2024-08-19T15:36:49.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490' 2024-08-19T15:36:49.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:36:49.904 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836490 2024-08-19T15:36:49.904 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:36:49.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:36:49.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836490 2024-08-19T15:36:49.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:36:49.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836490 2024-08-19T15:36:49.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-08-19T15:36:49.907 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836490 2024-08-19T15:36:49.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:36:50.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836490 2024-08-19T15:36:50.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:36:51.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:36:51.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:36:51.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836490 2024-08-19T15:36:51.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:36:51.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:36:51.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:36:51.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-08-19T15:36:51.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:36:51.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:36:51.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:36:51.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:36:51.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:36:51.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:36:51.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:36:52.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-08-19T15:36:52.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:36:52.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:36:52.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:36:52.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-08-19T15:36:52.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:36:52.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:36:52.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 5 = 1 ']' 2024-08-19T15:36:52.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-08-19T15:36:52.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 5 = 6 ']' 2024-08-19T15:36:52.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-08-19T15:36:52.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-6 1 1 2024-08-19T15:36:52.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-6 1 1 2024-08-19T15:36:52.788 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'psr_pool-6' created 2024-08-19T15:36:52.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:36:53.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-08-19T15:36:53.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:36:53.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:36:53.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:36:53.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:36:53.803 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:36:53.803 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:36:53.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:36:53.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:36:53.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:36:53.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:36:53.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:36:53.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:36:53.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:36:53.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:36:53.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:36:54.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-08-19T15:36:54.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:36:54.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:36:54.165 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:36:54.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836492 2024-08-19T15:36:54.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836492 2024-08-19T15:36:54.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492' 2024-08-19T15:36:54.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:36:54.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836492 2024-08-19T15:36:54.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:36:54.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:36:54.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836492 2024-08-19T15:36:54.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:36:54.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836492 2024-08-19T15:36:54.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-08-19T15:36:54.309 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836492 2024-08-19T15:36:54.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:36:54.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836492 2024-08-19T15:36:54.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:36:55.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:36:55.574 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:36:55.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836492 2024-08-19T15:36:55.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:36:55.845 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:36:55.845 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:36:56.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-08-19T15:36:56.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:36:56.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:36:56.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:36:56.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:36:56.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:36:56.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:36:56.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:36:56.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-08-19T15:36:56.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:36:56.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:36:56.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:36:56.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-08-19T15:36:56.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:36:56.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:36:56.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 6 = 1 ']' 2024-08-19T15:36:56.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-08-19T15:36:56.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 6 = 6 ']' 2024-08-19T15:36:56.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5868: TEST_scrub_warning: ceph osd pool set psr_pool-6 deep_scrub_interval 1209600 2024-08-19T15:36:57.181 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 6 deep_scrub_interval to 1209600 2024-08-19T15:36:57.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-08-19T15:36:57.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-7 1 1 2024-08-19T15:36:57.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-7 1 1 2024-08-19T15:36:57.557 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'psr_pool-7' created 2024-08-19T15:36:57.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:36:58.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-08-19T15:36:58.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:36:58.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:36:58.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:36:58.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:36:58.572 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:36:58.572 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:36:58.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:36:58.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:36:58.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:36:58.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:36:58.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:36:58.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:36:58.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:36:58.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:36:58.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:36:58.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-08-19T15:36:58.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:36:58.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:36:58.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:36:59.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836494 2024-08-19T15:36:59.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836494 2024-08-19T15:36:59.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494' 2024-08-19T15:36:59.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:36:59.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836494 2024-08-19T15:36:59.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:36:59.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:36:59.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836494 2024-08-19T15:36:59.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:36:59.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836494 2024-08-19T15:36:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-08-19T15:36:59.111 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836494 2024-08-19T15:36:59.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:36:59.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836494 2024-08-19T15:36:59.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:37:00.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:37:00.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:37:00.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836494 2024-08-19T15:37:00.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:37:01.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:37:01.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:37:01.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836494 2024-08-19T15:37:01.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:37:01.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:37:01.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:37:02.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 7 == 0 2024-08-19T15:37:02.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:37:02.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:37:02.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:37:02.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:37:02.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:37:02.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:37:02.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:37:02.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=7 2024-08-19T15:37:02.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:37:02.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:37:02.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:37:02.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 7 = 7 2024-08-19T15:37:02.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:37:02.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:37:02.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 7 = 1 ']' 2024-08-19T15:37:02.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-08-19T15:37:02.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 7 = 6 ']' 2024-08-19T15:37:02.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-08-19T15:37:02.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-8 1 1 2024-08-19T15:37:02.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-8 1 1 2024-08-19T15:37:03.285 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'psr_pool-8' created 2024-08-19T15:37:03.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:37:04.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-08-19T15:37:04.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:37:04.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:37:04.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:37:04.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:37:04.300 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:37:04.300 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:37:04.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:37:04.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:37:04.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:37:04.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:37:04.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:37:04.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:37:04.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:37:04.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:37:04.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:37:04.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-08-19T15:37:04.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:37:04.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:37:04.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:37:04.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836496 2024-08-19T15:37:04.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836496 2024-08-19T15:37:04.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496' 2024-08-19T15:37:04.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:37:04.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836496 2024-08-19T15:37:04.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:37:04.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:37:04.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836496 2024-08-19T15:37:04.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:37:04.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836496 2024-08-19T15:37:04.780 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836496 2024-08-19T15:37:04.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-08-19T15:37:04.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:37:05.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836496 2024-08-19T15:37:05.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:37:05.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:37:05.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:37:05.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-08-19T15:37:05.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:37:05.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:37:05.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:37:05.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:37:05.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:37:05.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:37:05.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:37:05.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-08-19T15:37:05.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:37:05.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:37:05.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:37:05.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-08-19T15:37:05.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:37:05.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:37:05.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 8 = 1 ']' 2024-08-19T15:37:05.979 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-08-19T15:37:05.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 8 = 6 ']' 2024-08-19T15:37:05.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-08-19T15:37:05.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-9 1 1 2024-08-19T15:37:05.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-9 1 1 2024-08-19T15:37:06.335 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'psr_pool-9' created 2024-08-19T15:37:06.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:37:07.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-08-19T15:37:07.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:37:07.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:37:07.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:37:07.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:37:07.349 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:37:07.349 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:37:07.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:37:07.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:37:07.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:37:07.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:37:07.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:37:07.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:37:07.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:37:07.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:37:07.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:37:07.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-08-19T15:37:07.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:37:07.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:37:07.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:37:07.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-08-19T15:37:07.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-08-19T15:37:07.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-08-19T15:37:07.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:37:07.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-08-19T15:37:07.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:37:07.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:37:07.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-08-19T15:37:07.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:37:07.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-08-19T15:37:07.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-08-19T15:37:07.829 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836498 2024-08-19T15:37:07.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:37:08.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-08-19T15:37:08.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:37:09.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:37:09.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:37:09.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-08-19T15:37:09.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:37:09.373 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:37:09.373 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:37:09.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-08-19T15:37:09.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:37:09.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:37:09.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:37:09.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:37:09.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:37:09.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:37:09.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:37:09.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-08-19T15:37:09.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:37:09.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:37:09.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:37:10.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-08-19T15:37:10.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:37:10.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:37:10.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 9 = 1 ']' 2024-08-19T15:37:10.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-08-19T15:37:10.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 9 = 6 ']' 2024-08-19T15:37:10.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-08-19T15:37:10.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-10 1 1 2024-08-19T15:37:10.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-10 1 1 2024-08-19T15:37:10.685 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'psr_pool-10' created 2024-08-19T15:37:10.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:37:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-08-19T15:37:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:37:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:37:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:37:11.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:37:11.700 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:37:11.700 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:37:11.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:37:11.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:37:11.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:37:11.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:37:11.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:37:11.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:37:11.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:37:11.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:37:11.806 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:37:12.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-08-19T15:37:12.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:37:12.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:37:12.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:37:12.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-08-19T15:37:12.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-08-19T15:37:12.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-08-19T15:37:12.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:37:12.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-08-19T15:37:12.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:37:12.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:37:12.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-08-19T15:37:12.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:37:12.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-08-19T15:37:12.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-08-19T15:37:12.178 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836500 2024-08-19T15:37:12.179 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:37:12.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836500 2024-08-19T15:37:12.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:37:13.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:37:13.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:37:13.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-08-19T15:37:13.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:37:13.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:37:13.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:37:14.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 10 == 0 2024-08-19T15:37:14.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:37:14.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:37:14.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:37:14.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:37:14.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:37:14.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:37:14.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:37:14.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=10 2024-08-19T15:37:14.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:37:14.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:37:14.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:37:14.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 10 = 10 2024-08-19T15:37:14.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:37:14.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:37:14.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 10 = 1 ']' 2024-08-19T15:37:14.666 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-08-19T15:37:14.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 10 = 6 ']' 2024-08-19T15:37:14.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5873: TEST_scrub_warning: local primary=0 2024-08-19T15:37:14.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5875: TEST_scrub_warning: ceph osd set noscrub 2024-08-19T15:37:15.045 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:37:15.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5876: TEST_scrub_warning: ceph osd set nodeep-scrub 2024-08-19T15:37:15.422 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:37:15.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5877: TEST_scrub_warning: ceph config set global osd_scrub_interval_randomize_ratio 0 2024-08-19T15:37:15.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5878: TEST_scrub_warning: ceph config set global osd_deep_scrub_randomize_ratio 0 2024-08-19T15:37:16.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: ceph config set global osd_scrub_max_interval 604800 2024-08-19T15:37:16.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5880: TEST_scrub_warning: ceph config set global osd_deep_scrub_interval 604800 2024-08-19T15:37:16.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: seq 1 5 2024-08-19T15:37:16.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-08-19T15:37:16.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5885: TEST_scrub_warning: '[' 1 = 1 ']' 2024-08-19T15:37:16.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: overdue_seconds=1900800 2024-08-19T15:37:16.816 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: expr 1900800 + 100 2024-08-19T15:37:16.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: ceph tell 1.0 schedule-scrub 1900900 2024-08-19T15:37:16.917 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:37:16.917 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": false, 2024-08-19T15:37:16.917 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:37:16.917 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-07-28T15:33:56.916738+0000" 2024-08-19T15:37:16.917 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:37:16.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-08-19T15:37:16.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5885: TEST_scrub_warning: '[' 2 = 1 ']' 2024-08-19T15:37:16.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5889: TEST_scrub_warning: overdue_seconds=993600 2024-08-19T15:37:16.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: expr 993600 + 200 2024-08-19T15:37:16.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: ceph tell 2.0 schedule-scrub 993800 2024-08-19T15:37:17.056 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:37:17.056 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": false, 2024-08-19T15:37:17.057 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:37:17.057 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-08T03:32:17.056555+0000" 2024-08-19T15:37:17.057 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:37:17.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-08-19T15:37:17.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5885: TEST_scrub_warning: '[' 3 = 1 ']' 2024-08-19T15:37:17.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5889: TEST_scrub_warning: overdue_seconds=993600 2024-08-19T15:37:17.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: expr 993600 + 300 2024-08-19T15:37:17.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: ceph tell 3.0 schedule-scrub 993900 2024-08-19T15:37:17.164 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:37:17.164 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": false, 2024-08-19T15:37:17.164 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:37:17.164 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-08T03:30:37.163881+0000" 2024-08-19T15:37:17.164 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:37:17.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-08-19T15:37:17.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5885: TEST_scrub_warning: '[' 4 = 1 ']' 2024-08-19T15:37:17.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5889: TEST_scrub_warning: overdue_seconds=993600 2024-08-19T15:37:17.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: expr 993600 + 400 2024-08-19T15:37:17.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: ceph tell 4.0 schedule-scrub 994000 2024-08-19T15:37:17.271 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:37:17.271 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": false, 2024-08-19T15:37:17.271 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:37:17.271 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-08T03:28:57.270983+0000" 2024-08-19T15:37:17.271 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:37:17.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-08-19T15:37:17.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5885: TEST_scrub_warning: '[' 5 = 1 ']' 2024-08-19T15:37:17.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5889: TEST_scrub_warning: overdue_seconds=993600 2024-08-19T15:37:17.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: expr 993600 + 500 2024-08-19T15:37:17.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: ceph tell 5.0 schedule-scrub 994100 2024-08-19T15:37:17.380 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:37:17.380 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": false, 2024-08-19T15:37:17.380 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:37:17.380 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-08T03:27:17.380053+0000" 2024-08-19T15:37:17.380 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:37:17.389 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 5 + 1 2024-08-19T15:37:17.391 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 5 + 5 2024-08-19T15:37:17.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: seq 6 10 2024-08-19T15:37:17.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-08-19T15:37:17.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: expr 5 + 1 2024-08-19T15:37:17.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: '[' 6 = 6 ']' 2024-08-19T15:37:17.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5898: TEST_scrub_warning: overdue_seconds=1900800 2024-08-19T15:37:17.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: expr 1900800 + 600 2024-08-19T15:37:17.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: ceph tell 6.0 schedule-deep-scrub 1901400 2024-08-19T15:37:17.492 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:37:17.492 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": true, 2024-08-19T15:37:17.492 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:37:17.493 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-07-28T15:25:37.492349+0000" 2024-08-19T15:37:17.493 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:37:17.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-08-19T15:37:17.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: expr 5 + 1 2024-08-19T15:37:17.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: '[' 7 = 6 ']' 2024-08-19T15:37:17.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5900: TEST_scrub_warning: overdue_seconds=993600 2024-08-19T15:37:17.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: expr 993600 + 700 2024-08-19T15:37:17.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: ceph tell 7.0 schedule-deep-scrub 994300 2024-08-19T15:37:17.600 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:37:17.601 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": true, 2024-08-19T15:37:17.601 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:37:17.601 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-08T03:23:57.600656+0000" 2024-08-19T15:37:17.601 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:37:17.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-08-19T15:37:17.610 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: expr 5 + 1 2024-08-19T15:37:17.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: '[' 8 = 6 ']' 2024-08-19T15:37:17.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5900: TEST_scrub_warning: overdue_seconds=993600 2024-08-19T15:37:17.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: expr 993600 + 800 2024-08-19T15:37:17.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: ceph tell 8.0 schedule-deep-scrub 994400 2024-08-19T15:37:17.710 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:37:17.710 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": true, 2024-08-19T15:37:17.710 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:37:17.710 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-08T03:22:17.709939+0000" 2024-08-19T15:37:17.710 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:37:17.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-08-19T15:37:17.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: expr 5 + 1 2024-08-19T15:37:17.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: '[' 9 = 6 ']' 2024-08-19T15:37:17.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5900: TEST_scrub_warning: overdue_seconds=993600 2024-08-19T15:37:17.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: expr 993600 + 900 2024-08-19T15:37:17.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: ceph tell 9.0 schedule-deep-scrub 994500 2024-08-19T15:37:17.820 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:37:17.820 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": true, 2024-08-19T15:37:17.820 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:37:17.820 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-08T03:20:37.819736+0000" 2024-08-19T15:37:17.820 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:37:17.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-08-19T15:37:17.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: expr 5 + 1 2024-08-19T15:37:17.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: '[' 10 = 6 ']' 2024-08-19T15:37:17.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5900: TEST_scrub_warning: overdue_seconds=993600 2024-08-19T15:37:17.830 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: expr 993600 + 1000 2024-08-19T15:37:17.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: ceph tell 10.0 schedule-deep-scrub 994600 2024-08-19T15:37:17.929 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:37:17.929 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": true, 2024-08-19T15:37:17.929 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:37:17.929 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-08T03:18:57.928948+0000" 2024-08-19T15:37:17.929 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:37:17.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5904: TEST_scrub_warning: flush_pg_stats 2024-08-19T15:37:17.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:37:17.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:37:18.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-08-19T15:37:18.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:37:18.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:37:18.207 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:37:18.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-08-19T15:37:18.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-08-19T15:37:18.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-08-19T15:37:18.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:37:18.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-08-19T15:37:18.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:37:18.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:37:18.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-08-19T15:37:18.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:37:18.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-08-19T15:37:18.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-08-19T15:37:18.314 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836502 2024-08-19T15:37:18.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:37:18.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-08-19T15:37:18.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:37:19.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:37:19.579 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:37:19.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-08-19T15:37:19.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5906: TEST_scrub_warning: ceph health 2024-08-19T15:37:20.144 INFO:tasks.workunit.client.0.smithi102.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 10 pgs not scrubbed in time; 10 pool(s) do not have an application enabled; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2024-08-19T15:37:20.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5907: TEST_scrub_warning: ceph health detail 2024-08-19T15:37:20.473 INFO:tasks.workunit.client.0.smithi102.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 10 pgs not scrubbed in time; 10 pool(s) do not have an application enabled; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2024-08-19T15:37:20.474 INFO:tasks.workunit.client.0.smithi102.stdout:[WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-08-19T15:37:20.474 INFO:tasks.workunit.client.0.smithi102.stdout:[WRN] PG_NOT_DEEP_SCRUBBED: 5 pgs not deep-scrubbed in time 2024-08-19T15:37:20.474 INFO:tasks.workunit.client.0.smithi102.stdout: pg 10.0 not deep-scrubbed since 2024-08-08T03:18:57.928948+0000 2024-08-19T15:37:20.474 INFO:tasks.workunit.client.0.smithi102.stdout: pg 9.0 not deep-scrubbed since 2024-08-08T03:20:37.819736+0000 2024-08-19T15:37:20.474 INFO:tasks.workunit.client.0.smithi102.stdout: pg 8.0 not deep-scrubbed since 2024-08-08T03:22:17.709939+0000 2024-08-19T15:37:20.474 INFO:tasks.workunit.client.0.smithi102.stdout: pg 7.0 not deep-scrubbed since 2024-08-08T03:23:57.600656+0000 2024-08-19T15:37:20.474 INFO:tasks.workunit.client.0.smithi102.stdout: pg 6.0 not deep-scrubbed since 2024-07-28T15:25:37.492349+0000 2024-08-19T15:37:20.474 INFO:tasks.workunit.client.0.smithi102.stdout:[WRN] PG_NOT_SCRUBBED: 10 pgs not scrubbed in time 2024-08-19T15:37:20.474 INFO:tasks.workunit.client.0.smithi102.stdout: pg 10.0 not scrubbed since 2024-08-08T03:18:57.928948+0000 2024-08-19T15:37:20.474 INFO:tasks.workunit.client.0.smithi102.stdout: pg 9.0 not scrubbed since 2024-08-08T03:20:37.819736+0000 2024-08-19T15:37:20.475 INFO:tasks.workunit.client.0.smithi102.stdout: pg 8.0 not scrubbed since 2024-08-08T03:22:17.709939+0000 2024-08-19T15:37:20.475 INFO:tasks.workunit.client.0.smithi102.stdout: pg 7.0 not scrubbed since 2024-08-08T03:23:57.600656+0000 2024-08-19T15:37:20.475 INFO:tasks.workunit.client.0.smithi102.stdout: pg 6.0 not scrubbed since 2024-07-28T15:25:37.492349+0000 2024-08-19T15:37:20.475 INFO:tasks.workunit.client.0.smithi102.stdout: pg 5.0 not scrubbed since 2024-08-08T03:27:17.380053+0000 2024-08-19T15:37:20.475 INFO:tasks.workunit.client.0.smithi102.stdout: pg 4.0 not scrubbed since 2024-08-08T03:28:57.270983+0000 2024-08-19T15:37:20.475 INFO:tasks.workunit.client.0.smithi102.stdout: pg 3.0 not scrubbed since 2024-08-08T03:30:37.163881+0000 2024-08-19T15:37:20.475 INFO:tasks.workunit.client.0.smithi102.stdout: pg 2.0 not scrubbed since 2024-08-08T03:32:17.056555+0000 2024-08-19T15:37:20.475 INFO:tasks.workunit.client.0.smithi102.stdout: pg 1.0 not scrubbed since 2024-07-28T15:33:56.916738+0000 2024-08-19T15:37:20.475 INFO:tasks.workunit.client.0.smithi102.stdout:[WRN] POOL_APP_NOT_ENABLED: 10 pool(s) do not have an application enabled 2024-08-19T15:37:20.475 INFO:tasks.workunit.client.0.smithi102.stdout: application not enabled on pool 'psr_pool-1' 2024-08-19T15:37:20.475 INFO:tasks.workunit.client.0.smithi102.stdout: application not enabled on pool 'psr_pool-2' 2024-08-19T15:37:20.476 INFO:tasks.workunit.client.0.smithi102.stdout: application not enabled on pool 'psr_pool-3' 2024-08-19T15:37:20.476 INFO:tasks.workunit.client.0.smithi102.stdout: application not enabled on pool 'psr_pool-4' 2024-08-19T15:37:20.476 INFO:tasks.workunit.client.0.smithi102.stdout: application not enabled on pool 'psr_pool-5' 2024-08-19T15:37:20.476 INFO:tasks.workunit.client.0.smithi102.stdout: application not enabled on pool 'psr_pool-6' 2024-08-19T15:37:20.476 INFO:tasks.workunit.client.0.smithi102.stdout: application not enabled on pool 'psr_pool-7' 2024-08-19T15:37:20.476 INFO:tasks.workunit.client.0.smithi102.stdout: application not enabled on pool 'psr_pool-8' 2024-08-19T15:37:20.476 INFO:tasks.workunit.client.0.smithi102.stdout: application not enabled on pool 'psr_pool-9' 2024-08-19T15:37:20.476 INFO:tasks.workunit.client.0.smithi102.stdout: application not enabled on pool 'psr_pool-10' 2024-08-19T15:37:20.476 INFO:tasks.workunit.client.0.smithi102.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-08-19T15:37:20.476 INFO:tasks.workunit.client.0.smithi102.stdout:[WRN] POOL_NO_REDUNDANCY: 10 pool(s) have no replicas configured 2024-08-19T15:37:20.476 INFO:tasks.workunit.client.0.smithi102.stdout: pool 'psr_pool-1' has no replicas configured 2024-08-19T15:37:20.476 INFO:tasks.workunit.client.0.smithi102.stdout: pool 'psr_pool-2' has no replicas configured 2024-08-19T15:37:20.476 INFO:tasks.workunit.client.0.smithi102.stdout: pool 'psr_pool-3' has no replicas configured 2024-08-19T15:37:20.477 INFO:tasks.workunit.client.0.smithi102.stdout: pool 'psr_pool-4' has no replicas configured 2024-08-19T15:37:20.477 INFO:tasks.workunit.client.0.smithi102.stdout: pool 'psr_pool-5' has no replicas configured 2024-08-19T15:37:20.477 INFO:tasks.workunit.client.0.smithi102.stdout: pool 'psr_pool-6' has no replicas configured 2024-08-19T15:37:20.477 INFO:tasks.workunit.client.0.smithi102.stdout: pool 'psr_pool-7' has no replicas configured 2024-08-19T15:37:20.477 INFO:tasks.workunit.client.0.smithi102.stdout: pool 'psr_pool-8' has no replicas configured 2024-08-19T15:37:20.477 INFO:tasks.workunit.client.0.smithi102.stdout: pool 'psr_pool-9' has no replicas configured 2024-08-19T15:37:20.477 INFO:tasks.workunit.client.0.smithi102.stdout: pool 'psr_pool-10' has no replicas configured 2024-08-19T15:37:20.477 INFO:tasks.workunit.client.0.smithi102.stdout:[WRN] TOO_FEW_OSDS: OSD count 1 < osd_pool_default_size 3 2024-08-19T15:37:20.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: TEST_scrub_warning: ceph health 2024-08-19T15:37:20.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: TEST_scrub_warning: grep -q ' pgs not deep-scrubbed in time' 2024-08-19T15:37:20.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5909: TEST_scrub_warning: ceph health 2024-08-19T15:37:20.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5909: TEST_scrub_warning: grep -q ' pgs not scrubbed in time' 2024-08-19T15:37:21.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5914: TEST_scrub_warning: ceph health detail 2024-08-19T15:37:21.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5914: TEST_scrub_warning: grep 'not scrubbed since' 2024-08-19T15:37:21.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5914: TEST_scrub_warning: wc -l 2024-08-19T15:37:21.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5914: TEST_scrub_warning: COUNT=10 2024-08-19T15:37:21.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5916: TEST_scrub_warning: expr 5+5 2024-08-19T15:37:21.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5916: TEST_scrub_warning: (( 10 != 5 && 10 != 5+5 )) 2024-08-19T15:37:21.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5920: TEST_scrub_warning: ceph health detail 2024-08-19T15:37:21.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5920: TEST_scrub_warning: grep 'not deep-scrubbed since' 2024-08-19T15:37:21.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5920: TEST_scrub_warning: wc -l 2024-08-19T15:37:21.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5920: TEST_scrub_warning: COUNT=5 2024-08-19T15:37:21.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5921: TEST_scrub_warning: '[' 5 '!=' 5 ']' 2024-08-19T15:37:21.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T15:37:21.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:37:21.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:37:21.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:37:21.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:37:21.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:37:21.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:37:21.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:37:21.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:37:21.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:37:21.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:37:21.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:37:21.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:37:21.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:37:21.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:37:21.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:37:21.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:37:21.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:37:21.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:37:21.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:37:21.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:37:21.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:37:21.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:37:21.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:37:21.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:37:21.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:37:21.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:37:21.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:37:21.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:37:21.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T15:37:21.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T15:37:21.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T15:37:21.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T15:37:21.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:37:21.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:37:21.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:37:21.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:37:21.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:37:21.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:37:21.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:37:21.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:37:21.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:37:21.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:37:21.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:37:21.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:37:21.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:37:21.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:37:21.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:37:21.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:37:21.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:37:21.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:37:21.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:37:21.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:37:21.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:37:21.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:37:21.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:37:21.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:37:21.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:37:21.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:37:21.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:37:21.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:37:21.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T15:37:21.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T15:37:21.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:37:21.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:37:21.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T15:37:21.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T15:37:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T15:37:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T15:37:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T15:37:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_skip_non_repair_during_recovery td/osd-scrub-repair 2024-08-19T15:37:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:136: TEST_skip_non_repair_during_recovery: local dir=td/osd-scrub-repair 2024-08-19T15:37:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:137: TEST_skip_non_repair_during_recovery: local poolname=rbd 2024-08-19T15:37:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:139: TEST_skip_non_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-08-19T15:37:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T15:37:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T15:37:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T15:37:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T15:37:21.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T15:37:21.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-08-19T15:37:22.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T15:37:22.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:37:22.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:37:22.275 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:37:22.275 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:37:22.275 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:37:22.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:37:22.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-08-19T15:37:22.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T15:37:22.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T15:37:22.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:37:22.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:37:22.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T15:37:22.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T15:37:22.452 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:37:22.452 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:37:22.452 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:37:22.453 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:37:22.453 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:37:22.453 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:37:22.453 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:37:22.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:37:22.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T15:37:22.534 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T15:37:22.534 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:37:22.534 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:37:22.534 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T15:37:22.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T15:37:22.535 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:37:22.535 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:37:22.535 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:37:22.535 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:37:22.535 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:37:22.535 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:37:22.535 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:37:22.536 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:37:22.536 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T15:37:22.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:140: TEST_skip_non_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-08-19T15:37:22.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T15:37:22.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T15:37:22.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T15:37:22.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T15:37:22.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T15:37:22.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T15:37:23.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T15:37:23.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:37:23.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:37:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:37:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:37:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:37:23.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:37:23.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:37:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:37:23.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:141: TEST_skip_non_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-08-19T15:37:23.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:37:23.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:37:23.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T15:37:23.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:37:23.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:37:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:37:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:37:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:37:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:37:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:37:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:37:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:37:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:37:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:37:23.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:37:23.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:37:23.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:37:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:37:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:37:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:37:23.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:37:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:37:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:37:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:37:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:37:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:37:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:37:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:37:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:37:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:37:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:37:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:37:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-08-19T15:37:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:37:23.466 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:37:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=87735eef-b0b2-45e5-867c-e4fff9b40682 2024-08-19T15:37:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 87735eef-b0b2-45e5-867c-e4fff9b40682' 2024-08-19T15:37:23.466 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 87735eef-b0b2-45e5-867c-e4fff9b40682 2024-08-19T15:37:23.466 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:37:23.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCzZsNmf4J8HBAA9Up5NrO9P/V0N2bxoctTzQ== 2024-08-19T15:37:23.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCzZsNmf4J8HBAA9Up5NrO9P/V0N2bxoctTzQ=="}' 2024-08-19T15:37:23.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 87735eef-b0b2-45e5-867c-e4fff9b40682 -i td/osd-scrub-repair/0/new.json 2024-08-19T15:37:23.700 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:37:23.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T15:37:23.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQCzZsNmf4J8HBAA9Up5NrO9P/V0N2bxoctTzQ== --osd-uuid 87735eef-b0b2-45e5-867c-e4fff9b40682 2024-08-19T15:37:23.741 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:37:23.739+0000 7f0c06415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:37:23.751 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:37:23.749+0000 7f0c06415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:37:23.752 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:37:23.751+0000 7f0c06415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:37:23.753 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:37:23.751+0000 7f0c06415640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:37:23.753 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:37:23.751+0000 7f0c06415640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:37:23.753 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:37:23.751+0000 7f0c06415640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:37:23.843 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:37:23.841+0000 7f0c06415640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T15:37:29.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T15:37:29.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:37:29.133 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T15:37:29.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T15:37:29.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:37:29.475 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:37:29.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T15:37:29.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:37:29.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-08-19T15:37:29.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:37:29.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:37:29.504 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:37:29.503+0000 7f36be74a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:37:29.506 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:37:29.505+0000 7f36be74a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:37:29.507 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:37:29.506+0000 7f36be74a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:37:29.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T15:37:29.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:37:29.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:37:29.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:37:29.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:37:29.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:37:29.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:37:29.742 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:37:29.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:37:29.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:37:30.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:37:30.032 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:37:30.031+0000 7f36be74a640 -1 Falling back to public interface 2024-08-19T15:37:31.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:37:31.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:37:31.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:37:31.006 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:37:31.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:37:31.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:37:31.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:37:32.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:37:32.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:37:32.277 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:37:32.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:37:32.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:37:32.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:37:32.351 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:37:32.350+0000 7f36be74a640 -1 osd.0 0 log_to_monitors true 2024-08-19T15:37:32.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:37:33.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:37:33.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:37:33.546 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:37:33.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:37:33.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:37:33.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:37:33.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:37:34.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:37:34.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:37:34.821 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:37:34.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:37:34.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:37:34.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:37:35.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:37:36.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:37:36.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:37:36.099 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:37:36.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:37:36.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:37:36.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:37:36.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:37:37.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:37:37.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:37:37.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:37:37.371 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:37:37.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:37:37.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:37:37.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:37:38.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:37:38.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:37:38.648 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:37:38.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:37:38.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:37:38.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:37:38.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:37:39.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:37:39.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:37:39.930 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:37:39.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:37:39.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:37:39.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:37:40.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:37:41.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:37:41.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:37:41.314 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:37:41.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:37:41.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:37:41.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:37:41.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:37:42.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:37:42.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:37:42.594 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:37:42.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:37:42.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:37:42.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:37:43.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:37:44.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:37:44.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:37:44.102 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:37:44.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:37:44.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:37:44.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:37:44.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:37:45.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:37:45.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:37:45.551 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:37:45.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:37:45.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:37:45.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:37:45.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:37:46.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:37:46.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:37:46.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:37:46.965 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:37:46.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:37:46.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:37:47.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:37:48.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:37:48.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:37:48.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:37:48.370 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:37:48.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:37:48.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:37:49.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:37:50.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:37:50.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:37:50.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:37:50.020 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:37:50.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:37:50.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:37:50.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:37:51.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:37:51.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:37:51.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:37:51.563 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:37:51.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:37:51.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:37:52.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:37:53.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:37:53.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:37:53.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:37:53.400 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:37:53.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:37:53.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:37:53.728 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1121839057,v1:127.0.0.1:6803/1121839057] [v2:127.0.0.1:6804/1121839057,v1:127.0.0.1:6805/1121839057] exists,up 87735eef-b0b2-45e5-867c-e4fff9b40682 2024-08-19T15:37:53.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:37:53.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:37:53.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:37:53.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:144: TEST_skip_non_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-08-19T15:37:53.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:37:53.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:37:53.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T15:37:53.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:37:53.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:37:53.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:37:53.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:37:53.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:37:53.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:37:53.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:37:53.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:37:53.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:37:53.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:37:53.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:37:53.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:37:53.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:37:53.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:37:53.733 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:37:53.733 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:37:53.733 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:37:53.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:37:53.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:37:53.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:37:53.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:37:53.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:37:53.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:37:53.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:37:53.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:37:53.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:37:53.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:37:53.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:37:53.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:37:53.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-08-19T15:37:53.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:37:53.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:37:53.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fda7c38c-4585-4078-b4f9-5bde305fe224 2024-08-19T15:37:53.735 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 fda7c38c-4585-4078-b4f9-5bde305fe224 2024-08-19T15:37:53.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 fda7c38c-4585-4078-b4f9-5bde305fe224' 2024-08-19T15:37:53.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:37:53.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDRZsNmNLWzLBAAhMs4mjXzNIQ72INHtIRtZg== 2024-08-19T15:37:53.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDRZsNmNLWzLBAAhMs4mjXzNIQ72INHtIRtZg=="}' 2024-08-19T15:37:53.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fda7c38c-4585-4078-b4f9-5bde305fe224 -i td/osd-scrub-repair/1/new.json 2024-08-19T15:37:54.068 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:37:54.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T15:37:54.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQDRZsNmNLWzLBAAhMs4mjXzNIQ72INHtIRtZg== --osd-uuid fda7c38c-4585-4078-b4f9-5bde305fe224 2024-08-19T15:37:54.111 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:37:54.110+0000 7f129a4fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:37:54.153 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:37:54.151+0000 7f129a4fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:37:54.153 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:37:54.152+0000 7f129a4fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:37:54.154 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:37:54.153+0000 7f129a4fd640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:37:54.154 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:37:54.153+0000 7f129a4fd640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:37:54.154 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:37:54.153+0000 7f129a4fd640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:37:54.227 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:37:54.226+0000 7f129a4fd640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T15:37:59.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T15:37:59.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:37:59.862 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T15:37:59.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T15:37:59.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:38:00.204 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:38:00.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T15:38:00.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-08-19T15:38:00.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:38:00.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:38:00.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:38:00.232 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:00.231+0000 7f1692bdc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:38:00.234 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:00.233+0000 7f1692bdc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:38:00.235 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:00.234+0000 7f1692bdc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:38:00.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T15:38:00.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:38:00.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:38:00.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:38:00.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:38:00.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:00.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:38:00.473 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:38:00.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:00.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:38:00.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:00.763 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:00.762+0000 7f1692bdc640 -1 Falling back to public interface 2024-08-19T15:38:01.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:01.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:01.737 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:38:01.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:38:01.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:01.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:38:02.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:03.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:03.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:03.018 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:38:03.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:38:03.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:03.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:38:03.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:03.076+0000 7f1692bdc640 -1 osd.1 0 log_to_monitors true 2024-08-19T15:38:03.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:04.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:04.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:04.302 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:38:04.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:38:04.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:04.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:38:04.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:05.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:05.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:05.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:38:05.580 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:38:05.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:05.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:38:05.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:06.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:06.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:06.856 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:38:06.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:38:06.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:06.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:38:07.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:08.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:08.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:08.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:38:08.139 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:38:08.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:08.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:38:08.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:09.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:09.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:09.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:38:09.500 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:38:09.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:09.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:38:09.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:10.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:10.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:10.815 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:38:10.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:38:10.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:10.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:38:11.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:12.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:12.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:12.095 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:38:12.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:38:12.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:12.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:38:12.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:13.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:13.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:13.375 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:38:13.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:38:13.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:13.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:38:13.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:14.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:14.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:14.698 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:38:14.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:38:14.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:14.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:38:15.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:16.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:16.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:16.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:38:16.252 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:38:16.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:16.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:38:16.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:17.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:17.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:17.544 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:38:17.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:38:17.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:17.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:38:18.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:19.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:19.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:19.288 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:38:19.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:38:19.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:19.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:38:19.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:20.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:20.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:20.649 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:38:20.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:38:20.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:20.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:38:21.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:22.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:22.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:22.145 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:38:22.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:38:22.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:22.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:38:22.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:23.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:23.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:23.823 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:38:23.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:38:23.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:23.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:38:24.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:25.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:25.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:25.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:38:25.182 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:38:25.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:25.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:38:25.526 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1081691805,v1:127.0.0.1:6811/1081691805] [v2:127.0.0.1:6812/1081691805,v1:127.0.0.1:6813/1081691805] exists,up fda7c38c-4585-4078-b4f9-5bde305fe224 2024-08-19T15:38:25.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:38:25.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:38:25.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:38:25.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:147: TEST_skip_non_repair_during_recovery: create_rbd_pool 2024-08-19T15:38:25.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T15:38:25.835 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-08-19T15:38:25.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T15:38:25.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T15:38:26.208 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' created 2024-08-19T15:38:26.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:38:27.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T15:38:29.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:148: TEST_skip_non_repair_during_recovery: wait_for_clean 2024-08-19T15:38:29.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:38:29.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:38:29.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:38:29.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:38:29.453 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:38:29.453 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:38:29.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:38:29.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:38:29.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:38:29.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:38:29.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:38:29.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:38:29.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:38:29.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:38:29.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:38:29.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:38:29.905 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:38:29.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:38:29.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:38:29.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:38:30.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836490 2024-08-19T15:38:30.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836490 2024-08-19T15:38:30.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490' 2024-08-19T15:38:30.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:38:30.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:38:30.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672965 2024-08-19T15:38:30.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672965 2024-08-19T15:38:30.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490 1-42949672965' 2024-08-19T15:38:30.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:38:30.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836490 2024-08-19T15:38:30.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:38:30.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:38:30.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836490 2024-08-19T15:38:30.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:38:30.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836490 2024-08-19T15:38:30.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-08-19T15:38:30.114 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836490 2024-08-19T15:38:30.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:38:30.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836490 2024-08-19T15:38:30.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:38:31.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:38:31.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:38:31.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836490 2024-08-19T15:38:31.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:38:31.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672965 2024-08-19T15:38:31.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:38:31.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:38:31.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672965 2024-08-19T15:38:31.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:38:31.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672965 2024-08-19T15:38:31.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-08-19T15:38:31.662 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672965 2024-08-19T15:38:31.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:38:31.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672965 2024-08-19T15:38:31.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:38:31.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:38:31.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:38:32.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T15:38:32.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:38:32.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:38:32.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:38:32.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:38:32.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:38:32.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:38:32.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:38:32.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:38:32.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:38:32.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:38:32.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:38:32.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T15:38:32.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:38:32.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:38:32.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:150: TEST_skip_non_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-08-19T15:38:32.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:38:32.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-08-19T15:38:32.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-08-19T15:38:32.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:38:32.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:38:32.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:38:33.188 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:38:33.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:38:33.633 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:38:33.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:38:33.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:38:33.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-08-19T15:38:33.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:151: TEST_skip_non_repair_during_recovery: get_not_primary rbd SOMETHING 2024-08-19T15:38:33.880 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-08-19T15:38:33.880 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-08-19T15:38:33.880 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-08-19T15:38:33.880 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-08-19T15:38:33.880 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T15:38:33.880 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-08-19T15:38:33.880 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:38:34.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T15:38:34.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-08-19T15:38:34.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T15:38:34.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:151: TEST_skip_non_repair_during_recovery: scrub_and_not_schedule td/osd-scrub-repair rbd 0 2024-08-19T15:38:34.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:155: scrub_and_not_schedule: local dir=td/osd-scrub-repair 2024-08-19T15:38:34.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:156: scrub_and_not_schedule: local poolname=rbd 2024-08-19T15:38:34.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:157: scrub_and_not_schedule: local osd=0 2024-08-19T15:38:34.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:162: scrub_and_not_schedule: get_pg rbd SOMETHING 2024-08-19T15:38:34.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-08-19T15:38:34.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T15:38:34.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-08-19T15:38:34.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:38:34.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:162: scrub_and_not_schedule: local pg=1.3 2024-08-19T15:38:34.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:163: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-08-19T15:38:34.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:38:34.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:38:34.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:38:34.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:38:34.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:163: scrub_and_not_schedule: local last_scrub=2024-08-19T15:38:26.119026+0000 2024-08-19T15:38:34.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:164: scrub_and_not_schedule: ceph tell 1.3 schedule-scrub 2024-08-19T15:38:35.056 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:38:35.056 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": false, 2024-08-19T15:38:35.056 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:38:35.056 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-12T15:36:55.056390+0000" 2024-08-19T15:38:35.057 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:38:35.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i=0 )) 2024-08-19T15:38:35.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-08-19T15:38:35.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-08-19T15:38:35.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:38:35.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:38:35.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:38:35.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:38:35.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: test 2024-08-19T15:38:26.119026+0000 '>' 2024-08-19T15:38:26.119026+0000 2024-08-19T15:38:35.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:173: scrub_and_not_schedule: sleep 1 2024-08-19T15:38:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i++ )) 2024-08-19T15:38:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-08-19T15:38:36.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-08-19T15:38:36.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:38:36.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:38:36.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:38:36.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:38:36.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: test 2024-08-19T15:38:26.119026+0000 '>' 2024-08-19T15:38:26.119026+0000 2024-08-19T15:38:36.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:173: scrub_and_not_schedule: sleep 1 2024-08-19T15:38:37.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i++ )) 2024-08-19T15:38:37.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-08-19T15:38:37.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-08-19T15:38:37.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-08-19T15:38:37.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:38:37.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:38:37.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-08-19T15:38:37.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: test 2024-08-12T15:36:55.056390+0000 '>' 2024-08-19T15:38:26.119026+0000 2024-08-19T15:38:37.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:173: scrub_and_not_schedule: sleep 1 2024-08-19T15:38:38.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i++ )) 2024-08-19T15:38:38.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-08-19T15:38:38.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:179: scrub_and_not_schedule: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-08-19T15:38:38.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:38:38.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:38:38.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:38:38.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:38:38.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-08-19T15:38:38.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:38:38.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:38:38.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:38:38.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:38:38.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:38:38.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:38:38.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:38:38.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:38:38.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:38:38.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:38:39.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:38:39.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-08-19T15:38:39.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:38:39.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:38:39.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:38:39.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:38:39.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:38:39.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-08-19T15:38:40.449 INFO:tasks.workunit.client.0.smithi102.stdout:_ 2024-08-19T15:38:40.449 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 2024-08-19T15:38:40.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:38:40.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:38:40.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:38:40.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:38:40.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:38:40.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:38:40.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:38:40.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:38:40.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:38:40.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:38:40.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:38:40.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:38:40.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:38:40.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:38:40.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:38:40.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:38:40.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:38:40.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:38:40.718 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:38:40.719 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:38:40.719 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:38:40.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:38:40.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:38:40.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:38:40.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:38:40.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:38:40.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:38:40.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:38:40.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:38:40.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:38:40.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:38:40.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:38:40.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:38:40.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:38:40.721 INFO:tasks.workunit.client.0.smithi102.stderr:start osd.0 2024-08-19T15:38:40.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:38:40.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:38:40.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:38:40.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:38:40.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:38:40.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:38:40.748 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:40.747+0000 7fceaf40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:38:40.750 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:40.749+0000 7fceaf40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:38:40.752 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:40.751+0000 7fceaf40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:38:40.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:38:40.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:38:40.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:38:40.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:38:40.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:38:40.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:40.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:38:40.998 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:38:40.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:40.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:38:41.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:41.276 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:41.276+0000 7fceaf40d640 -1 Falling back to public interface 2024-08-19T15:38:42.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:42.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:42.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:38:42.268 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:38:42.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:42.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:38:42.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:43.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:43.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:43.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:38:43.541 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:38:43.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:43.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:38:43.654 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:43.653+0000 7fceaf40d640 -1 osd.0 19 log_to_monitors true 2024-08-19T15:38:43.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:44.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:44.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:44.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:38:44.890 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-08-19T15:38:44.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:44.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:38:45.171 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 up in weight 1 up_from 23 up_thru 23 down_at 20 last_clean_interval [5,19) [v2:127.0.0.1:6802/4053408411,v1:127.0.0.1:6803/4053408411] [v2:127.0.0.1:6804/4053408411,v1:127.0.0.1:6805/4053408411] exists,up 87735eef-b0b2-45e5-867c-e4fff9b40682 2024-08-19T15:38:45.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:38:45.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:38:45.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:38:45.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:38:45.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:38:45.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:38:45.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:38:45.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:38:45.172 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:38:45.172 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:38:45.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:38:45.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:38:45.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:38:45.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:38:45.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:38:45.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:38:45.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:38:45.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:38:45.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:38:45.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:38:45.608 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:38:45.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:38:45.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:38:45.609 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:38:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247810 2024-08-19T15:38:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247810 2024-08-19T15:38:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247810' 2024-08-19T15:38:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:38:45.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:38:45.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-08-19T15:38:45.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-08-19T15:38:45.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247810 1-42949672970' 2024-08-19T15:38:45.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:38:45.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247810 2024-08-19T15:38:45.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:38:45.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:38:45.820 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247810 2024-08-19T15:38:45.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:38:45.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247810 2024-08-19T15:38:45.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2024-08-19T15:38:45.822 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.0 seq 98784247810 2024-08-19T15:38:45.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:38:46.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 98784247810 2024-08-19T15:38:46.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:38:47.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:38:47.091 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:38:47.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 98784247810 2024-08-19T15:38:47.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:38:48.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:38:48.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:38:48.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247810 -lt 98784247810 2024-08-19T15:38:48.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:38:48.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-08-19T15:38:48.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:38:48.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:38:48.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-08-19T15:38:48.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:38:48.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-08-19T15:38:48.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-08-19T15:38:48.642 INFO:tasks.workunit.client.0.smithi102.stderr:waiting osd.1 seq 42949672970 2024-08-19T15:38:48.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:38:48.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-08-19T15:38:48.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:38:48.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:38:48.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:38:49.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T15:38:49.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:38:49.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:38:49.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:38:49.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:38:49.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:38:49.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:38:49.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:38:49.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T15:38:49.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:38:49.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:38:49.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:38:49.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T15:38:49.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:38:49.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:38:49.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:180: scrub_and_not_schedule: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-08-19T15:38:49.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:181: scrub_and_not_schedule: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-08-19T15:38:49.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T15:38:49.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:38:49.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:38:49.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:38:49.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:38:49.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:38:49.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:38:49.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:38:49.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:38:50.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:38:50.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:38:50.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:38:50.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:38:50.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:38:50.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:38:50.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:38:50.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:38:50.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:38:50.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:38:50.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:38:50.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:38:50.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:38:50.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:38:50.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:38:50.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:38:50.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:38:50.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:38:50.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:38:50.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:38:50.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T15:38:50.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T15:38:50.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T15:38:50.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T15:38:50.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:38:50.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:38:50.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:38:50.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:38:50.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:38:50.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:38:50.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:38:50.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:38:50.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:38:50.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:38:50.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:38:50.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:38:50.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:38:50.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:38:50.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:38:50.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:38:50.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:38:50.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:38:50.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:38:50.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:38:50.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:38:50.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:38:50.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:38:50.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:38:50.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:38:50.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:38:50.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:38:50.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:38:50.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T15:38:50.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T15:38:50.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:38:50.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:38:50.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T15:38:50.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T15:38:50.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T15:38:50.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T15:38:50.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T15:38:50.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_unfound_erasure_coded_appends td/osd-scrub-repair 2024-08-19T15:38:50.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:903: TEST_unfound_erasure_coded_appends: unfound_erasure_coded td/osd-scrub-repair false 2024-08-19T15:38:50.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: unfound_erasure_coded: local dir=td/osd-scrub-repair 2024-08-19T15:38:50.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:853: unfound_erasure_coded: local allow_overwrites=false 2024-08-19T15:38:50.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:854: unfound_erasure_coded: local poolname=ecpool 2024-08-19T15:38:50.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:855: unfound_erasure_coded: local payload=ABCDEF 2024-08-19T15:38:50.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2024-08-19T15:38:50.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T15:38:50.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T15:38:50.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T15:38:50.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T15:38:50.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T15:38:50.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-08-19T15:38:50.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T15:38:50.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:38:50.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:38:50.412 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:38:50.412 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:38:50.412 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:38:50.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:38:50.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T15:38:50.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T15:38:50.579 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T15:38:50.579 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:38:50.579 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:38:50.579 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T15:38:50.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T15:38:50.581 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:38:50.581 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:38:50.581 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:38:50.581 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:38:50.581 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:38:50.581 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:38:50.581 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:38:50.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:38:50.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T15:38:50.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T15:38:50.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:38:50.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:38:50.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T15:38:50.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T15:38:50.662 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:38:50.663 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:38:50.663 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:38:50.666 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:38:50.666 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:38:50.666 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:38:50.666 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:38:50.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:38:50.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T15:38:50.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2024-08-19T15:38:50.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T15:38:50.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T15:38:50.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T15:38:50.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T15:38:50.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T15:38:50.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T15:38:51.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T15:38:51.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:38:51.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:38:51.556 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:38:51.556 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:38:51.557 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:38:51.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:38:51.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:38:51.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:38:51.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: seq 0 3 2024-08-19T15:38:51.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-08-19T15:38:51.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2024-08-19T15:38:51.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:38:51.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:38:51.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T15:38:51.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:38:51.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:38:51.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:38:51.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:38:51.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:38:51.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:38:51.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:38:51.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:38:51.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:38:51.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:38:51.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:38:51.592 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:38:51.592 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:38:51.592 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:38:51.592 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:38:51.593 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:38:51.593 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:38:51.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:38:51.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:38:51.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:38:51.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:38:51.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:38:51.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:38:51.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:38:51.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:38:51.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:38:51.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:38:51.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:38:51.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:38:51.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:38:51.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:38:51.594 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:38:51.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4652248b-b97c-4a8d-b951-f7b9377ac255 2024-08-19T15:38:51.595 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 4652248b-b97c-4a8d-b951-f7b9377ac255 2024-08-19T15:38:51.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4652248b-b97c-4a8d-b951-f7b9377ac255' 2024-08-19T15:38:51.595 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:38:51.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALZ8NmTYNTJBAAff6nDtyHp44wYaumEtJ+JA== 2024-08-19T15:38:51.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALZ8NmTYNTJBAAff6nDtyHp44wYaumEtJ+JA=="}' 2024-08-19T15:38:51.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4652248b-b97c-4a8d-b951-f7b9377ac255 -i td/osd-scrub-repair/0/new.json 2024-08-19T15:38:51.836 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:38:51.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T15:38:51.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALZ8NmTYNTJBAAff6nDtyHp44wYaumEtJ+JA== --osd-uuid 4652248b-b97c-4a8d-b951-f7b9377ac255 2024-08-19T15:38:51.876 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:51.875+0000 7fec661ba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:38:51.887 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:51.886+0000 7fec661ba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:38:51.888 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:51.887+0000 7fec661ba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:38:51.888 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:51.887+0000 7fec661ba640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:38:51.888 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:51.887+0000 7fec661ba640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:38:51.889 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:51.887+0000 7fec661ba640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:38:51.979 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:51.977+0000 7fec661ba640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T15:38:57.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T15:38:57.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:38:57.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T15:38:57.489 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T15:38:57.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:38:57.835 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:38:57.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T15:38:57.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:38:57.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:38:57.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:38:57.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:38:57.862 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:57.860+0000 7fefcb933640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:38:57.863 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:57.862+0000 7fefcb933640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:38:57.865 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:57.863+0000 7fefcb933640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:38:58.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T15:38:58.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:38:58.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:38:58.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:38:58.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:38:58.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:58.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:38:58.105 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:38:58.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:58.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:38:58.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:38:58.384 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:38:58.382+0000 7fefcb933640 -1 Falling back to public interface 2024-08-19T15:38:59.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:38:59.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:38:59.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:38:59.370 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:38:59.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:38:59.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:38:59.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:00.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:00.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:00.643 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:39:00.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:39:00.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:00.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:39:00.701 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:00.699+0000 7fefcb933640 -1 osd.0 0 log_to_monitors true 2024-08-19T15:39:00.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:01.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:01.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:01.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:39:01.963 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:39:01.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:01.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:39:02.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:03.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:03.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:03.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:39:03.245 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:39:03.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:03.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:39:03.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:04.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:04.614 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:39:04.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:04.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:39:04.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:04.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:39:04.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:05.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:05.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:05.895 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:39:05.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:39:05.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:05.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:39:06.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:07.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:07.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:07.170 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:39:07.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:39:07.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:07.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:39:07.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:08.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:08.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:08.446 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:39:08.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:39:08.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:08.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:39:08.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:09.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:09.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:09.728 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:39:09.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:39:09.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:09.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:39:10.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:11.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:11.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:11.004 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:39:11.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:39:11.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:11.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:39:11.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:12.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:12.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:12.283 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:39:12.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:39:12.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:12.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:39:12.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:13.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:13.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:13.842 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:39:13.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:39:13.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:13.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:39:14.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:15.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:15.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:15.349 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:39:15.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:39:15.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:15.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:39:16.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:17.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:17.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:17.006 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:39:17.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:39:17.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:17.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:39:17.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:18.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:18.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:18.653 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:39:18.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:39:18.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:18.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:39:19.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:20.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:20.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:20.144 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:39:20.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:39:20.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:20.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:39:20.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:21.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:21.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:21.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:39:21.821 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:39:21.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:21.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:39:22.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:23.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:23.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:23.140 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:39:23.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:39:23.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:23.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:39:23.411 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4260768488,v1:127.0.0.1:6803/4260768488] [v2:127.0.0.1:6804/4260768488,v1:127.0.0.1:6805/4260768488] exists,up 4652248b-b97c-4a8d-b951-f7b9377ac255 2024-08-19T15:39:23.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:39:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:39:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:39:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-08-19T15:39:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2024-08-19T15:39:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:39:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:39:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T15:39:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:39:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:39:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:39:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:39:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:39:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:39:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:39:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:39:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:39:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:39:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:39:23.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:39:23.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:39:23.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:39:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:39:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:39:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:39:23.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:39:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:39:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:39:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:39:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:39:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:39:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:39:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:39:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:39:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:39:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:39:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:39:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:39:23.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:39:23.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:39:23.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e886c294-54a4-4400-9330-b0005229235d 2024-08-19T15:39:23.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e886c294-54a4-4400-9330-b0005229235d' 2024-08-19T15:39:23.419 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 e886c294-54a4-4400-9330-b0005229235d 2024-08-19T15:39:23.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:39:23.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQArZ8Nm3SXVGRAA/Ih3mtoVCX4fW6Bqn3yTPg== 2024-08-19T15:39:23.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQArZ8Nm3SXVGRAA/Ih3mtoVCX4fW6Bqn3yTPg=="}' 2024-08-19T15:39:23.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e886c294-54a4-4400-9330-b0005229235d -i td/osd-scrub-repair/1/new.json 2024-08-19T15:39:23.833 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:39:23.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T15:39:23.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQArZ8Nm3SXVGRAA/Ih3mtoVCX4fW6Bqn3yTPg== --osd-uuid e886c294-54a4-4400-9330-b0005229235d 2024-08-19T15:39:23.873 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:23.872+0000 7f971640e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:39:23.917 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:23.916+0000 7f971640e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:39:23.918 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:23.917+0000 7f971640e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:39:23.919 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:23.918+0000 7f971640e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:39:23.919 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:23.918+0000 7f971640e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:39:23.919 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:23.918+0000 7f971640e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:39:23.992 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:23.991+0000 7f971640e640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T15:39:29.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T15:39:29.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:39:29.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T15:39:29.589 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T15:39:29.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:39:29.951 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:39:29.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T15:39:29.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:39:29.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:39:29.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:39:29.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:39:29.978 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:29.977+0000 7f8218816640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:39:29.980 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:29.979+0000 7f8218816640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:39:29.981 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:29.980+0000 7f8218816640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:39:30.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T15:39:30.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:39:30.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:39:30.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:39:30.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:39:30.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:30.220 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:39:30.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:39:30.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:30.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:39:30.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:30.503 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:30.501+0000 7f8218816640 -1 Falling back to public interface 2024-08-19T15:39:31.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:31.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:31.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:39:31.485 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:39:31.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:31.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:39:31.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:32.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:32.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:32.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:39:32.773 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:39:32.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:32.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:39:32.824 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:32.823+0000 7f8218816640 -1 osd.1 0 log_to_monitors true 2024-08-19T15:39:33.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:34.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:34.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:34.050 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:39:34.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:39:34.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:34.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:39:34.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:35.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:35.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:35.490 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:39:35.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:39:35.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:35.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:39:36.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:37.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:37.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:37.005 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:39:37.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:39:37.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:37.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:39:37.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:38.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:38.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:38.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:39:38.283 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:39:38.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:38.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:39:38.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:39.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:39.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:39.656 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:39:39.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:39:39.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:39.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:39:39.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:40.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:40.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:40.932 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:39:40.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:39:40.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:40.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:39:41.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:42.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:42.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:42.213 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:39:42.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:39:42.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:42.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:39:42.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:43.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:43.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:43.491 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:39:43.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:39:43.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:43.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:39:43.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:44.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:44.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:44.905 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:39:44.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:39:44.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:44.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:39:45.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:46.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:46.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:46.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:39:46.179 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:39:46.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:46.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:39:46.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:47.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:47.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:47.874 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:39:47.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:39:47.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:47.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:39:48.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:49.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:49.558 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:39:49.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:49.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:39:49.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:49.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:39:50.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:51.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:51.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:51.286 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:39:51.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:39:51.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:51.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:39:51.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:52.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:52.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:52.867 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:39:52.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:39:52.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:52.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:39:53.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:54.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:54.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:54.230 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:39:54.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:39:54.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:54.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:39:54.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:39:55.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:39:55.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:39:55.670 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:39:55.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:39:55.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:39:55.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:39:55.975 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3677681448,v1:127.0.0.1:6811/3677681448] [v2:127.0.0.1:6812/3677681448,v1:127.0.0.1:6813/3677681448] exists,up e886c294-54a4-4400-9330-b0005229235d 2024-08-19T15:39:55.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:39:55.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:39:55.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:39:55.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-08-19T15:39:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2024-08-19T15:39:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:39:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:39:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T15:39:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:39:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:39:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:39:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:39:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:39:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:39:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T15:39:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T15:39:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:39:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:39:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:39:55.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:39:55.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:39:55.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:39:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:39:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:39:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:39:55.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:39:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:39:55.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:39:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:39:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:39:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:39:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:39:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:39:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:39:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:39:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:39:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:39:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:39:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T15:39:55.982 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:39:55.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f49476b4-350d-41d6-907d-1232d07fa3d5 2024-08-19T15:39:55.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f49476b4-350d-41d6-907d-1232d07fa3d5' 2024-08-19T15:39:55.983 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 f49476b4-350d-41d6-907d-1232d07fa3d5 2024-08-19T15:39:55.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:39:55.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBLZ8NmumpvOxAAlVsyUVm8bSwgybNYw9AIEg== 2024-08-19T15:39:55.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBLZ8NmumpvOxAAlVsyUVm8bSwgybNYw9AIEg=="}' 2024-08-19T15:39:55.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f49476b4-350d-41d6-907d-1232d07fa3d5 -i td/osd-scrub-repair/2/new.json 2024-08-19T15:39:56.369 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:39:56.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-08-19T15:39:56.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBLZ8NmumpvOxAAlVsyUVm8bSwgybNYw9AIEg== --osd-uuid f49476b4-350d-41d6-907d-1232d07fa3d5 2024-08-19T15:39:56.410 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:56.409+0000 7f08b0e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:39:56.446 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:56.445+0000 7f08b0e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:39:56.447 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:56.446+0000 7f08b0e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:39:56.447 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:56.446+0000 7f08b0e0d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:39:56.447 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:56.446+0000 7f08b0e0d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:39:56.447 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:56.446+0000 7f08b0e0d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:39:56.520 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:39:56.519+0000 7f08b0e0d640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-08-19T15:40:01.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-08-19T15:40:01.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:40:01.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T15:40:01.904 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T15:40:01.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:40:02.255 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T15:40:02.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T15:40:02.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:40:02.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:40:02.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:40:02.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:40:02.281 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:40:02.280+0000 7fe533843640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:40:02.283 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:40:02.282+0000 7fe533843640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:40:02.284 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:40:02.283+0000 7fe533843640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:40:02.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T15:40:02.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:40:02.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T15:40:02.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:40:02.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:40:02.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:02.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:40:02.522 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:40:02.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:02.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:02.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:02.821 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:40:02.820+0000 7fe533843640 -1 Falling back to public interface 2024-08-19T15:40:03.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:03.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:03.789 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:40:03.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:40:03.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:03.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:04.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:05.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:05.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:05.060 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:40:05.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:40:05.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:05.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:05.137 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:40:05.136+0000 7fe533843640 -1 osd.2 0 log_to_monitors true 2024-08-19T15:40:05.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:06.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:06.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:06.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:40:06.380 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:40:06.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:06.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:06.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:07.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:07.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:07.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:40:07.652 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:40:07.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:07.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:07.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:08.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:08.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:08.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:40:08.958 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:40:08.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:08.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:09.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:10.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:10.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:10.234 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:40:10.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:40:10.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:10.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:10.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:11.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:11.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:11.511 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:40:11.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:40:11.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:11.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:11.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:12.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:12.799 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:40:12.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:12.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:40:12.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:12.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:13.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:14.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:14.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:14.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:40:14.081 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:40:14.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:14.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:14.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:15.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:15.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:15.364 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:40:15.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:40:15.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:15.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:15.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:16.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:16.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:16.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:40:16.640 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:40:16.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:16.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:17.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:18.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:18.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:18.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:40:18.166 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:40:18.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:18.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:18.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:19.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:19.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:19.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:40:19.617 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:40:19.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:19.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:20.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:21.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:21.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:21.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:40:21.249 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:40:21.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:21.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:21.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:22.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:22.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:22.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:40:22.757 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:40:22.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:22.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:23.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:24.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:24.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:24.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:40:24.174 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:40:24.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:24.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:24.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:25.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:25.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:25.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:40:25.707 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:40:25.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:25.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:25.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:26.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:26.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:26.981 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:40:26.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:40:26.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:26.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:27.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:28.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:28.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:28.374 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T15:40:28.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T15:40:28.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:28.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:40:28.643 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2197212680,v1:127.0.0.1:6819/2197212680] [v2:127.0.0.1:6820/2197212680,v1:127.0.0.1:6821/2197212680] exists,up f49476b4-350d-41d6-907d-1232d07fa3d5 2024-08-19T15:40:28.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:40:28.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:40:28.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:40:28.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-08-19T15:40:28.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2024-08-19T15:40:28.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:40:28.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:40:28.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-08-19T15:40:28.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:40:28.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T15:40:28.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:40:28.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:40:28.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:40:28.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:40:28.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T15:40:28.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T15:40:28.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:40:28.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:40:28.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:40:28.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:40:28.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:40:28.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:40:28.649 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:40:28.649 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:40:28.649 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:40:28.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:40:28.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:40:28.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:40:28.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:40:28.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:40:28.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:40:28.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:40:28.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:40:28.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:40:28.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:40:28.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:40:28.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:40:28.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:40:28.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T15:40:28.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:40:28.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ebb6fb6e-9a60-4420-8582-0c54caed9d0a 2024-08-19T15:40:28.651 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 ebb6fb6e-9a60-4420-8582-0c54caed9d0a 2024-08-19T15:40:28.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 ebb6fb6e-9a60-4420-8582-0c54caed9d0a' 2024-08-19T15:40:28.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:40:28.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBsZ8NmqVeqJxAAJw9JjI3a2paQ8x8HJA3Dzw== 2024-08-19T15:40:28.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBsZ8NmqVeqJxAAJw9JjI3a2paQ8x8HJA3Dzw=="}' 2024-08-19T15:40:28.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ebb6fb6e-9a60-4420-8582-0c54caed9d0a -i td/osd-scrub-repair/3/new.json 2024-08-19T15:40:29.017 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:40:29.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-08-19T15:40:29.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBsZ8NmqVeqJxAAJw9JjI3a2paQ8x8HJA3Dzw== --osd-uuid ebb6fb6e-9a60-4420-8582-0c54caed9d0a 2024-08-19T15:40:29.058 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:40:29.057+0000 7f4fafbb1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:40:29.102 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:40:29.101+0000 7f4fafbb1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:40:29.103 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:40:29.102+0000 7f4fafbb1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:40:29.104 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:40:29.103+0000 7f4fafbb1640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-08-19T15:40:29.104 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:40:29.103+0000 7f4fafbb1640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-08-19T15:40:29.104 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:40:29.103+0000 7f4fafbb1640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-08-19T15:40:29.177 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:40:29.176+0000 7f4fafbb1640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-08-19T15:40:34.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-08-19T15:40:34.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:40:34.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-08-19T15:40:34.957 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-08-19T15:40:34.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:40:35.301 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-08-19T15:40:35.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-08-19T15:40:35.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:40:35.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:40:35.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:40:35.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:40:35.328 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:40:35.326+0000 7f8f8c8d5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:40:35.330 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:40:35.328+0000 7f8f8c8d5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:40:35.331 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:40:35.329+0000 7f8f8c8d5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:40:35.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-08-19T15:40:35.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:40:35.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T15:40:35.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:40:35.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:40:35.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:35.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:40:35.572 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:40:35.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:35.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:40:35.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:35.855 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:40:35.853+0000 7f8f8c8d5640 -1 Falling back to public interface 2024-08-19T15:40:36.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:36.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:36.844 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:40:36.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:40:36.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:36.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:40:37.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:38.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:38.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:38.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:40:38.123 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:40:38.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:38.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:40:38.211 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:40:38.209+0000 7f8f8c8d5640 -1 osd.3 0 log_to_monitors true 2024-08-19T15:40:38.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:39.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:39.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:39.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:40:39.421 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:40:39.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:39.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:40:39.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:40.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:40.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:40.706 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:40:40.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:40:40.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:40.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:40:41.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:42.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:42.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:42.027 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:40:42.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:40:42.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:42.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:40:42.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:43.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:43.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:43.355 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:40:43.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:40:43.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:43.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:40:43.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:44.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:44.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:44.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:40:44.631 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:40:44.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:44.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:40:44.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:45.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:45.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:45.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:40:45.911 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:40:45.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:45.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:40:46.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:47.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:47.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:47.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:40:47.187 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:40:47.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:47.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:40:47.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:48.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:48.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:48.461 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:40:48.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:40:48.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:48.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:40:48.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:49.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:49.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:49.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:40:49.740 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:40:49.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:49.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:40:50.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:51.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:51.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:51.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:40:51.480 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:40:51.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:51.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:40:51.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:52.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:52.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:52.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:40:52.983 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:40:52.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:52.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:40:53.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:54.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:54.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:54.504 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:40:54.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:40:54.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:54.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:40:55.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:56.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:56.332 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:40:56.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:56.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:40:56.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:56.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:40:56.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:57.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:57.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:57.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:40:57.859 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:40:57.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:57.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:40:58.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:40:59.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:40:59.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:40:59.285 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:40:59.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:40:59.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:40:59.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:40:59.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:00.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:00.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:00.712 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:41:00.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:41:00.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:00.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:41:01.065 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3263203809,v1:127.0.0.1:6827/3263203809] [v2:127.0.0.1:6828/3263203809,v1:127.0.0.1:6829/3263203809] exists,up ebb6fb6e-9a60-4420-8582-0c54caed9d0a 2024-08-19T15:41:01.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:41:01.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:41:01.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:41:01.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: unfound_erasure_coded: create_ec_pool ecpool false k=2 m=2 2024-08-19T15:41:01.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-08-19T15:41:01.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-08-19T15:41:01.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-08-19T15:41:01.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-08-19T15:41:01.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-08-19T15:41:01.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-08-19T15:41:01.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-08-19T15:41:02.255 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'ecpool' created 2024-08-19T15:41:02.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:41:03.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-08-19T15:41:03.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-08-19T15:41:03.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:41:03.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:41:03.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:41:03.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:41:03.270 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:41:03.270 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:41:03.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:41:03.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:41:03.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:41:03.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:41:03.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:41:03.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:41:03.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:41:03.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:41:03.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:41:03.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:41:03.645 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:41:03.645 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:41:03.645 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-08-19T15:41:03.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:41:03.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:41:03.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:41:03.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-08-19T15:41:03.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-08-19T15:41:03.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-08-19T15:41:03.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:41:03.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:41:03.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-08-19T15:41:03.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-08-19T15:41:03.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672978' 2024-08-19T15:41:03.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:41:03.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:41:04.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-08-19T15:41:04.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-08-19T15:41:04.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672978 2-64424509456' 2024-08-19T15:41:04.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:41:04.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:41:04.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378630 2024-08-19T15:41:04.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378630 2024-08-19T15:41:04.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672978 2-64424509456 3-81604378630' 2024-08-19T15:41:04.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:41:04.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-08-19T15:41:04.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:41:04.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:41:04.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-08-19T15:41:04.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:41:04.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-08-19T15:41:04.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-08-19T15:41:04.146 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836505 2024-08-19T15:41:04.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:41:04.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-08-19T15:41:04.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:41:05.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:41:05.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:41:05.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-08-19T15:41:05.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:41:05.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-08-19T15:41:05.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:41:05.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:41:05.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-08-19T15:41:05.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:41:05.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-08-19T15:41:05.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-08-19T15:41:05.713 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672978 2024-08-19T15:41:05.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:41:05.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-08-19T15:41:05.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:41:05.982 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-08-19T15:41:05.982 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:41:05.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:41:05.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-08-19T15:41:05.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:41:05.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-08-19T15:41:05.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-08-19T15:41:05.985 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509456 2024-08-19T15:41:05.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:41:06.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-08-19T15:41:06.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:41:06.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-81604378630 2024-08-19T15:41:06.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:41:06.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:41:06.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-81604378630 2024-08-19T15:41:06.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:41:06.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378630 2024-08-19T15:41:06.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 81604378630' 2024-08-19T15:41:06.254 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 81604378630 2024-08-19T15:41:06.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:41:06.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378630 -lt 81604378630 2024-08-19T15:41:06.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:41:06.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:41:06.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:41:06.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:41:06.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:41:06.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:41:06.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:41:06.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:41:06.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:41:06.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:41:06.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:41:07.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-19T15:41:07.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:41:07.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:41:07.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:41:07.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-19T15:41:07.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-19T15:41:07.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T15:41:07.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-19T15:41:07.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T15:41:07.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T15:41:07.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T15:41:07.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:41:07.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:41:07.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:41:07.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:41:07.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:41:07.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:41:07.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:41:07.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:41:07.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:41:07.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:41:07.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:41:08.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:41:08.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:41:08.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:41:08.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-08-19T15:41:08.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-08-19T15:41:08.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:41:08.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:41:08.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-08-19T15:41:08.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:41:08.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:41:08.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:41:08.506 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:41:08.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:41:08.900 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:41:08.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:41:08.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:41:08.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-08-19T15:41:09.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: get_primary ecpool SOMETHING 2024-08-19T15:41:09.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-08-19T15:41:09.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T15:41:09.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-08-19T15:41:09.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:41:09.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: local primary=1 2024-08-19T15:41:09.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: get_osds ecpool SOMETHING 2024-08-19T15:41:09.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: sed -e s/1// 2024-08-19T15:41:09.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-08-19T15:41:09.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-08-19T15:41:09.505 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-08-19T15:41:09.505 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-19T15:41:09.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-08-19T15:41:09.784 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:41:09.784 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:41:09.784 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-08-19T15:41:09.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 3 2024-08-19T15:41:09.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: osds=('0' '2' '3') 2024-08-19T15:41:09.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: local -a osds 2024-08-19T15:41:09.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: local not_primary_first=0 2024-08-19T15:41:09.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: unfound_erasure_coded: local not_primary_second=2 2024-08-19T15:41:09.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:871: unfound_erasure_coded: local not_primary_third=3 2024-08-19T15:41:09.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:876: unfound_erasure_coded: pids= 2024-08-19T15:41:09.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T15:41:09.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T15:41:09.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T15:41:09.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 453977"' 2024-08-19T15:41:09.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 453977' 2024-08-19T15:41:09.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:878: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-08-19T15:41:09.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T15:41:09.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T15:41:09.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 453979"' 2024-08-19T15:41:09.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T15:41:09.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 453979' 2024-08-19T15:41:09.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:879: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T15:41:09.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T15:41:09.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/453980: /' 2024-08-19T15:41:09.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T15:41:09.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 453982"' 2024-08-19T15:41:09.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 453982' 2024-08-19T15:41:09.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-08-19T15:41:09.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:880: unfound_erasure_coded: wait_background pids 2024-08-19T15:41:09.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 453977 453979 453982' 2024-08-19T15:41:09.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-08-19T15:41:09.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T15:41:09.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 453977 2024-08-19T15:41:09.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/453984: /' 2024-08-19T15:41:09.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T15:41:09.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/453986: /' 2024-08-19T15:41:12.528 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:41:12.528 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:41:12.528 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:41:12.528 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:41:12.528 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T15:41:12.528 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:41:12.528 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:41:12.528 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:41:12.528 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:41:12.528 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:41:12.529 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:41:12.529 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:41:12.529 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:41:12.529 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:41:12.529 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:41:12.529 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:41:12.529 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T15:41:12.529 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:41:12.529 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:41:12.529 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:41:12.530 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:41:12.530 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:41:12.530 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-08-19T15:41:12.530 INFO:tasks.workunit.client.0.smithi102.stderr:453980: remove 1#1:eb822e21:::SOMETHING:head# 2024-08-19T15:41:12.530 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:41:12.530 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:41:12.530 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:41:12.530 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:41:12.530 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:41:12.530 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:41:12.531 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:41:12.531 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:41:12.531 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:41:12.532 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:41:12.532 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:41:12.532 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:41:12.532 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:41:12.532 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:41:12.532 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:41:12.532 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:41:12.532 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:41:12.532 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:41:12.532 INFO:tasks.workunit.client.0.smithi102.stderr:453980: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:41:12.532 INFO:tasks.workunit.client.0.smithi102.stderr:453980: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:41:12.533 INFO:tasks.workunit.client.0.smithi102.stderr:453980: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:41:12.533 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:41:12.533 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:41:12.533 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:41:12.533 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:41:12.533 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:41:12.533 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:41:12.533 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:41:12.533 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:41:12.533 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:41:12.534 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:41:12.536 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:41:12.536 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:41:12.536 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:41:12.536 INFO:tasks.workunit.client.0.smithi102.stderr:453980: start osd.0 2024-08-19T15:41:12.536 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:41:12.537 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:41:12.537 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:41:12.537 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-08-19T15:41:12.537 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:41:12.537 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-08-19T15:41:12.537 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:41:12.537 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:41:12.537 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-08-19T15:41:12.537 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:41:12.537 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-08-19T15:41:12.537 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:41:12.537 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:41:12.538 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:41:12.538 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:41:12.538 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:41:12.538 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:41:12.538 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-08-19T15:41:12.538 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:41:12.538 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:41:12.538 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-08-19T15:41:12.538 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:41:12.538 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:41:12.538 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-08-19T15:41:12.538 INFO:tasks.workunit.client.0.smithi102.stderr:453984: remove 2#1:eb822e21:::SOMETHING:head# 2024-08-19T15:41:12.538 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-08-19T15:41:12.539 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:41:12.539 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:41:12.539 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-19T15:41:12.539 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:41:12.539 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:41:12.539 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:41:12.539 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:41:12.539 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:41:12.539 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:41:12.539 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T15:41:12.539 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T15:41:12.539 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:41:12.539 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:41:12.540 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:41:12.540 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:41:12.540 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:41:12.540 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:41:12.540 INFO:tasks.workunit.client.0.smithi102.stderr:453984: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:41:12.540 INFO:tasks.workunit.client.0.smithi102.stderr:453984: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:41:12.540 INFO:tasks.workunit.client.0.smithi102.stderr:453984: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:41:12.540 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:41:12.540 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:41:12.540 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:41:12.540 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:41:12.540 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:41:12.540 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:41:12.541 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:41:12.541 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:41:12.541 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:41:12.541 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:41:12.541 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:41:12.541 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T15:41:12.541 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-19T15:41:12.541 INFO:tasks.workunit.client.0.smithi102.stderr:453984: start osd.2 2024-08-19T15:41:12.541 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:41:12.571 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:41:12.571 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:41:12.571 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-08-19T15:41:12.572 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:41:12.572 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T15:41:12.572 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:41:12.572 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:41:12.572 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-08-19T15:41:12.572 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:41:12.572 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-08-19T15:41:12.572 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:41:12.572 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:41:12.572 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:41:12.572 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:41:12.572 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:41:12.572 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:41:12.573 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T15:41:12.573 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:41:12.573 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:41:12.573 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-08-19T15:41:12.573 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:41:12.573 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-08-19T15:41:12.573 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-08-19T15:41:12.573 INFO:tasks.workunit.client.0.smithi102.stderr:453986: remove 3#1:eb822e21:::SOMETHING:head# 2024-08-19T15:41:12.573 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-08-19T15:41:12.573 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:41:12.573 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:41:12.573 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-19T15:41:12.573 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:41:12.574 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T15:41:12.574 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:41:12.574 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:41:12.575 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:41:12.575 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:41:12.575 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T15:41:12.575 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T15:41:12.575 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:41:12.575 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:41:12.576 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:41:12.576 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:41:12.576 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:41:12.576 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:41:12.576 INFO:tasks.workunit.client.0.smithi102.stderr:453986: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:41:12.576 INFO:tasks.workunit.client.0.smithi102.stderr:453986: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:41:12.576 INFO:tasks.workunit.client.0.smithi102.stderr:453986: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:41:12.576 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:41:12.576 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:41:12.576 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:41:12.576 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:41:12.576 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:41:12.577 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:41:12.577 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:41:12.577 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:41:12.577 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:41:12.577 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:41:12.577 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:41:12.577 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T15:41:12.577 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-19T15:41:12.577 INFO:tasks.workunit.client.0.smithi102.stderr:453986: start osd.3 2024-08-19T15:41:12.577 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:41:17.101 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-08-19T15:41:17.101 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-19T15:41:17.101 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:41:17.101 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:41:17.101 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:41:17.101 INFO:tasks.workunit.client.0.smithi102.stderr:453984: 2024-08-19T15:41:12.565+0000 7f633420d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:41:17.101 INFO:tasks.workunit.client.0.smithi102.stderr:453984: 2024-08-19T15:41:12.567+0000 7f633420d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:41:17.101 INFO:tasks.workunit.client.0.smithi102.stderr:453984: 2024-08-19T15:41:12.569+0000 7f633420d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:41:17.101 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-19T15:41:17.102 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:41:17.102 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T15:41:17.102 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:41:17.102 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:41:17.102 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:17.102 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:41:17.102 INFO:tasks.workunit.client.0.smithi102.stderr:453984: 0 2024-08-19T15:41:17.102 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:17.102 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:41:17.102 INFO:tasks.workunit.client.0.smithi102.stderr:453984: 2024-08-19T15:41:13.093+0000 7f633420d640 -1 Falling back to public interface 2024-08-19T15:41:17.102 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:17.102 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:17.103 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:17.103 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:41:17.103 INFO:tasks.workunit.client.0.smithi102.stderr:453984: 1 2024-08-19T15:41:17.103 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:17.103 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:41:17.103 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:17.103 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:17.103 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:17.103 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:41:17.103 INFO:tasks.workunit.client.0.smithi102.stderr:453984: 2 2024-08-19T15:41:17.104 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:17.104 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:41:17.104 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:17.104 INFO:tasks.workunit.client.0.smithi102.stderr:453984: 2024-08-19T15:41:16.031+0000 7f633420d640 -1 osd.2 27 log_to_monitors true 2024-08-19T15:41:17.104 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:17.104 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:17.104 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:41:17.104 INFO:tasks.workunit.client.0.smithi102.stderr:453984: 3 2024-08-19T15:41:17.104 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:17.104 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:41:17.104 INFO:tasks.workunit.client.0.smithi102.stderr:453984: osd.2 up in weight 1 up_from 32 up_thru 0 down_at 28 last_c453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:41:17.104 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:41:17.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:41:17.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:41:17.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:41:17.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: 2024-08-19T15:41:12.560+0000 7f4e19c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:41:17.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: 2024-08-19T15:41:12.562+0000 7f4e19c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:41:17.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: 2024-08-19T15:41:12.564+0000 7f4e19c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:41:17.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:41:17.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:41:17.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:41:17.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:41:17.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:41:17.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:17.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:41:17.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: 0 2024-08-19T15:41:17.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:17.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:17.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: 2024-08-19T15:41:13.088+0000 7f4e19c18640 -1 Falling back to public interface 2024-08-19T15:41:17.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:17.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:17.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:17.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:41:17.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: 1 2024-08-19T15:41:17.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:17.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:17.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:17.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:17.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:17.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:41:17.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: 2 2024-08-19T15:41:17.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:17.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:17.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:17.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: 2024-08-19T15:41:15.931+0000 7f4e19c18640 -1 osd.0 27 log_to_monitors true 2024-08-19T15:41:17.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:17.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:17.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:41:17.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: 3 2024-08-19T15:41:17.108 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:17.108 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:17.234 INFO:tasks.workunit.client.0.smithi102.stderr:453980: osd.0 up in weight 1 up_from 31 up_thru 0 down_at 28 last_c453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-08-19T15:41:17.234 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-19T15:41:17.234 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:41:17.235 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:41:17.235 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:41:17.235 INFO:tasks.workunit.client.0.smithi102.stderr:453986: 2024-08-19T15:41:12.603+0000 7f1ef805f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:41:17.235 INFO:tasks.workunit.client.0.smithi102.stderr:453986: 2024-08-19T15:41:12.605+0000 7f1ef805f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:41:17.235 INFO:tasks.workunit.client.0.smithi102.stderr:453986: 2024-08-19T15:41:12.606+0000 7f1ef805f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:41:17.235 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-19T15:41:17.235 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:41:17.235 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T15:41:17.235 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:41:17.235 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:41:17.235 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:17.235 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:41:17.235 INFO:tasks.workunit.client.0.smithi102.stderr:453986: 0 2024-08-19T15:41:17.236 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:17.236 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:41:17.236 INFO:tasks.workunit.client.0.smithi102.stderr:453986: 2024-08-19T15:41:13.138+0000 7f1ef805f640 -1 Falling back to public interface 2024-08-19T15:41:17.236 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:17.236 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:17.236 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:17.236 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:41:17.236 INFO:tasks.workunit.client.0.smithi102.stderr:453986: 1 2024-08-19T15:41:17.236 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:17.236 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:41:17.236 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:17.236 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:17.236 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:17.237 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:41:17.237 INFO:tasks.workunit.client.0.smithi102.stderr:453986: 2 2024-08-19T15:41:17.237 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:17.237 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:41:17.237 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:17.237 INFO:tasks.workunit.client.0.smithi102.stderr:453986: 2024-08-19T15:41:16.082+0000 7f1ef805f640 -1 osd.3 27 log_to_monitors true 2024-08-19T15:41:17.237 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:17.237 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:17.237 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:41:17.237 INFO:tasks.workunit.client.0.smithi102.stderr:453986: 3 2024-08-19T15:41:17.237 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:17.237 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:41:17.939 INFO:tasks.workunit.client.0.smithi102.stderr:453986: osd.3 up in weight 1 up_from 32 up_thru 0 down_at 28 last_clean_interval [15,27) [v2:127.0.0.1:6818/1967113034,v1:127.0.0.1:6819/1967113034] [v2:127.0.0.1:6820/1967113034,v1:127.0.0.1:6821/1967113034] exists,up f49476b4-350d-41d6-907d-1232d07fa3d5 2024-08-19T15:41:17.939 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:41:17.940 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:41:17.940 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:41:17.940 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:41:17.940 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:41:17.940 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:41:17.940 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:41:17.940 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:41:17.940 INFO:tasks.workunit.client.0.smithi102.stderr:453984: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:41:17.940 INFO:tasks.workunit.client.0.smithi102.stderr:453984: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:41:17.940 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:41:17.940 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:41:17.940 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:41:17.941 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:41:17.941 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:41:17.941 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:41:17.941 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:41:17.941 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:41:17.941 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:41:17.941 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:41:17.941 INFO:tasks.workunit.client.0.smithi102.stderr:453984: 1 2024-08-19T15:41:17.941 INFO:tasks.workunit.client.0.smithi102.stderr:453984: 2 2024-08-19T15:41:17.941 INFO:tasks.workunit.client.0.smithi102.stderr:453984: 3' 2024-08-19T15:41:17.941 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:41:17.941 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:41:17.941 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:41:17.941 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=133143986178 2024-08-19T15:41:17.942 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 133143986178 2024-08-19T15:41:17.942 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-133143986178' 2024-08-19T15:41:17.942 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:41:17.942 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:41:17.942 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-08-19T15:41:17.942 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-08-19T15:41:17.942 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-133143986178 1-42949672983' 2024-08-19T15:41:17.942 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:41:17.942 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph lean_interval [5,27) [v2:127.0.0.1:6802/238082918,v1:127.0.0.1:6803/238082918] [v2:127.0.0.1:6804/238082918,v1:127.0.0.1:6805/238082918] exists,up 4652248b-b97c-4a8d-b951-f7b9377ac255 2024-08-19T15:41:17.942 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:41:17.942 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:41:17.943 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:41:17.943 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:41:17.943 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:41:17.943 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:41:17.943 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:41:17.943 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:41:17.943 INFO:tasks.workunit.client.0.smithi102.stderr:453980: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:41:17.943 INFO:tasks.workunit.client.0.smithi102.stderr:453980: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:41:17.943 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:41:17.943 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:41:17.943 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:41:17.943 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:41:17.943 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:41:17.944 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:41:17.944 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:41:17.944 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:41:17.944 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:41:17.944 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:41:17.944 INFO:tasks.workunit.client.0.smithi102.stderr:453980: 1 2024-08-19T15:41:17.944 INFO:tasks.workunit.client.0.smithi102.stderr:453980: 2 2024-08-19T15:41:17.944 INFO:tasks.workunit.client.0.smithi102.stderr:453980: 3' 2024-08-19T15:41:17.944 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:41:17.944 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:41:17.944 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:41:17.944 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=133143986179 2024-08-19T15:41:17.944 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 133143986179 2024-08-19T15:41:17.944 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-133143986179' 2024-08-19T15:41:17.945 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:41:17.945 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:41:17.945 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-08-19T15:41:17.945 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-08-19T15:41:17.945 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-133143986179 1-42949672982' 2024-08-19T15:41:17.945 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:41:18.054 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell lean_interval [19,27) [v2:127.0.0.1:6826/529519115,v1:127.0.0.1:6827/529519115] [v2:127.0.0.1:6828/529519115,v1:127.0.0.1:6829/529519115] exists,up ebb6fb6e-9a60-4420-8582-0c54caed9d0a 2024-08-19T15:41:18.055 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:41:18.055 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:41:18.055 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:41:18.055 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:41:18.055 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:41:18.055 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:41:18.055 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:41:18.055 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:41:18.055 INFO:tasks.workunit.client.0.smithi102.stderr:453986: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:41:18.055 INFO:tasks.workunit.client.0.smithi102.stderr:453986: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:41:18.055 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:41:18.056 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:41:18.056 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:41:18.056 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:41:18.056 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:41:18.056 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:41:18.056 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:41:18.056 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:41:18.056 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:41:18.056 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:41:18.056 INFO:tasks.workunit.client.0.smithi102.stderr:453986: 1 2024-08-19T15:41:18.056 INFO:tasks.workunit.client.0.smithi102.stderr:453986: 2 2024-08-19T15:41:18.056 INFO:tasks.workunit.client.0.smithi102.stderr:453986: 3' 2024-08-19T15:41:18.056 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:41:18.057 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:41:18.057 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:41:18.057 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=133143986180 2024-08-19T15:41:18.057 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 133143986180 2024-08-19T15:41:18.057 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-133143986180' 2024-08-19T15:41:18.057 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:41:18.057 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:41:18.057 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672984 2024-08-19T15:41:18.057 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672984 2024-08-19T15:41:18.057 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-133143986180 1-42949672984' 2024-08-19T15:41:18.057 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:41:20.104 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tellosd.2 flush_pg_stats 2024-08-19T15:41:20.104 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-08-19T15:41:20.104 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-08-19T15:41:20.104 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-133143986179 1-42949672982 2-137438953474' 2024-08-19T15:41:20.104 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:41:20.104 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:41:20.104 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-08-19T15:41:20.104 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-08-19T15:41:20.104 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-133143986179 1-42949672982 2-137438953474 3-137438953474' 2024-08-19T15:41:20.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:41:20.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-133143986179 2024-08-19T15:41:20.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:41:20.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:41:20.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-133143986179 2024-08-19T15:41:20.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:41:20.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=133143986179 2024-08-19T15:41:20.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 133143986179' 2024-08-19T15:41:20.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: waiting osd.0 seq 133143986179 2024-08-19T15:41:20.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:41:20.105 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 133143986179 2024-08-19T15:41:20.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:41:20.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:41:20.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:41:20.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 133143986180 -lt 133143986179 2024-08-19T15:41:20.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:41:20.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-08-19T15:41:20.106 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:41:20.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:41:20.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-08-19T15:41:20.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:41:20.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-08-19T15:41:20.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-08-19T15:41:20.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: waiting osd.1 seq 42949672982 2024-08-19T15:41:20.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:41:20.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672982 2024-08-19T15:41:20.107 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:41:20.108 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubutell osd.2 flush_pg_stats 2024-08-19T15:41:20.109 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953475 2024-08-19T15:41:20.109 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953475 2024-08-19T15:41:20.109 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-133143986178 1-42949672983 2-137438953475' 2024-08-19T15:41:20.109 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:41:20.109 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:41:20.109 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953475 2024-08-19T15:41:20.109 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953475 2024-08-19T15:41:20.109 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-133143986178 1-42949672983 2-137438953475 3-137438953475' 2024-08-19T15:41:20.109 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:41:20.109 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-133143986178 2024-08-19T15:41:20.109 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:41:20.109 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:41:20.109 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-133143986178 2024-08-19T15:41:20.109 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:41:20.110 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=133143986178 2024-08-19T15:41:20.110 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 133143986178' 2024-08-19T15:41:20.110 INFO:tasks.workunit.client.0.smithi102.stderr:453984: waiting osd.0 seq 133143986178 2024-08-19T15:41:20.110 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:41:20.110 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 133143986178 2024-08-19T15:41:20.110 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:41:20.110 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:41:20.110 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:41:20.110 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 133143986180 -lt 133143986178 2024-08-19T15:41:20.110 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:41:20.111 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-08-19T15:41:20.111 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:41:20.111 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:41:20.111 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-08-19T15:41:20.111 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:41:20.111 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-08-19T15:41:20.111 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-08-19T15:41:20.111 INFO:tasks.workunit.client.0.smithi102.stderr:453984: waiting osd.1 seq 42949672983 2024-08-19T15:41:20.111 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:41:20.111 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672983 2024-08-19T15:41:20.111 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:41:20.224 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //hom osd.2 flush_pg_stats 2024-08-19T15:41:20.225 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953476 2024-08-19T15:41:20.225 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953476 2024-08-19T15:41:20.225 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-133143986180 1-42949672984 2-137438953476' 2024-08-19T15:41:20.225 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:41:20.225 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:41:20.225 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953476 2024-08-19T15:41:20.225 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953476 2024-08-19T15:41:20.225 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-133143986180 1-42949672984 2-137438953476 3-137438953476' 2024-08-19T15:41:20.225 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:41:20.225 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-133143986180 2024-08-19T15:41:20.225 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:41:20.226 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:41:20.226 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-133143986180 2024-08-19T15:41:20.226 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:41:20.226 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=133143986180 2024-08-19T15:41:20.226 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 133143986180' 2024-08-19T15:41:20.226 INFO:tasks.workunit.client.0.smithi102.stderr:453986: waiting osd.0 seq 133143986180 2024-08-19T15:41:20.226 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:41:20.226 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 133143986180 2024-08-19T15:41:20.226 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:41:20.226 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:41:20.226 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:41:20.227 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 133143986180 -lt 133143986180 2024-08-19T15:41:20.227 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:41:20.227 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672984 2024-08-19T15:41:20.227 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:41:20.227 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:41:20.227 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672984 2024-08-19T15:41:20.227 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:41:20.228 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672984 2024-08-19T15:41:20.228 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-08-19T15:41:20.228 INFO:tasks.workunit.client.0.smithi102.stderr:453986: waiting osd.1 seq 42949672984 2024-08-19T15:41:20.228 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:41:20.228 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672984 2024-08-19T15:41:20.228 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:41:21.396 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953474 2024-08-19T15:41:21.396 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:41:21.396 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:41:21.396 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953474 2024-08-19T15:41:21.397 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:41:21.397 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-08-19T15:41:21.397 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953474' 2024-08-19T15:41:21.397 INFO:tasks.workunit.client.0.smithi102.stderr:453980: waiting osd.2 seq 137438953474 2024-08-19T15:41:21.397 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:41:21.397 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953476 -lt 137438953474 2024-08-19T15:41:21.397 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:41:21.397 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-137438953474 2024-08-19T15:41:21.397 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:41:21.397 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:41:21.397 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-137438953474 2024-08-19T15:41:21.397 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:41:21.398 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-08-19T15:41:21.398 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 137438953474' 2024-08-19T15:41:21.398 INFO:tasks.workunit.client.0.smithi102.stderr:453980: waiting osd.3 seq 137438953474 2024-08-19T15:41:21.398 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:41:21.398 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953476 -lt 137438953474 2024-08-19T15:41:21.398 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:41:21.398 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:41:21.398 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:41:21.398 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:41:21.398 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:41:21.398 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:41:21.398 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:41:21.399 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:41:21.399 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:41:21.399 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:41:21.399 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:41:21.399 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:41:21.399 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:41:21.401 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/stae/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953475 2024-08-19T15:41:21.401 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:41:21.401 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:41:21.401 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953475 2024-08-19T15:41:21.402 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:41:21.402 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953475 2024-08-19T15:41:21.402 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953475' 2024-08-19T15:41:21.402 INFO:tasks.workunit.client.0.smithi102.stderr:453984: waiting osd.2 seq 137438953475 2024-08-19T15:41:21.402 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:41:21.402 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953476 -lt 137438953475 2024-08-19T15:41:21.403 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:41:21.403 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-137438953475 2024-08-19T15:41:21.403 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:41:21.403 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:41:21.403 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-137438953475 2024-08-19T15:41:21.403 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:41:21.403 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953475 2024-08-19T15:41:21.403 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 137438953475' 2024-08-19T15:41:21.403 INFO:tasks.workunit.client.0.smithi102.stderr:453984: waiting osd.3 seq 137438953475 2024-08-19T15:41:21.404 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:41:21.404 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953476 -lt 137438953475 2024-08-19T15:41:21.404 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:41:21.404 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:41:21.404 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:41:21.404 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:41:21.404 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:41:21.404 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:41:21.405 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:41:21.405 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:41:21.405 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:41:21.405 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:41:21.405 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:41:21.405 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:41:21.405 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:41:21.545 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/quntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953476 2024-08-19T15:41:21.545 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:41:21.545 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:41:21.546 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953476 2024-08-19T15:41:21.546 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:41:21.546 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953476 2024-08-19T15:41:21.546 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953476' 2024-08-19T15:41:21.546 INFO:tasks.workunit.client.0.smithi102.stderr:453986: waiting osd.2 seq 137438953476 2024-08-19T15:41:21.546 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:41:21.546 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953476 -lt 137438953476 2024-08-19T15:41:21.546 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:41:21.546 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-137438953476 2024-08-19T15:41:21.546 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:41:21.546 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:41:21.546 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-137438953476 2024-08-19T15:41:21.547 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:41:21.547 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953476 2024-08-19T15:41:21.547 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 137438953476' 2024-08-19T15:41:21.547 INFO:tasks.workunit.client.0.smithi102.stderr:453986: waiting osd.3 seq 137438953476 2024-08-19T15:41:21.547 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:41:21.547 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953476 -lt 137438953476 2024-08-19T15:41:21.547 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:41:21.547 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:41:21.547 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:41:21.547 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:41:21.547 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:41:21.547 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:41:21.547 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:41:21.548 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:41:21.548 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:41:21.548 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:41:21.548 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:41:21.548 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:41:21.548 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:41:21.745 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/stndalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:41:21.746 INFO:tasks.workunit.client.0.smithi102.stderr:453980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:41:21.746 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:41:21.746 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:41:21.746 INFO:tasks.workunit.client.0.smithi102.stderr:453980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:41:21.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T15:41:21.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T15:41:21.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 453979 2024-08-19T15:41:21.749 INFO:tasks.workunit.client.0.smithi102.stderr:a/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:41:21.749 INFO:tasks.workunit.client.0.smithi102.stderr:453984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:41:21.749 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:41:21.750 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:41:21.750 INFO:tasks.workunit.client.0.smithi102.stderr:453984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:41:21.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T15:41:21.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T15:41:21.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 453982 2024-08-19T15:41:21.886 INFO:tasks.workunit.client.0.smithi102.stderr:andalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:41:21.886 INFO:tasks.workunit.client.0.smithi102.stderr:453986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:41:21.887 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:41:21.887 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:41:21.887 INFO:tasks.workunit.client.0.smithi102.stderr:453986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:41:21.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T15:41:21.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-08-19T15:41:21.887 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-08-19T15:41:21.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-08-19T15:41:21.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: return_code=0 2024-08-19T15:41:21.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:882: unfound_erasure_coded: '[' 0 -ne 0 ']' 2024-08-19T15:41:21.887 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: get_pg ecpool SOMETHING 2024-08-19T15:41:21.887 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T15:41:21.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T15:41:21.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T15:41:21.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:41:22.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: local pg=1.0 2024-08-19T15:41:22.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: unfound_erasure_coded: repair 1.0 2024-08-19T15:41:22.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-08-19T15:41:22.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-08-19T15:41:22.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:41:22.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:41:22.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:41:22.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:41:22.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T15:41:02.137038+0000 2024-08-19T15:41:22.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-08-19T15:41:22.678 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-08-19T15:41:22.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-08-19T15:41:02.137038+0000 2024-08-19T15:41:22.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-08-19T15:41:22.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:41:02.137038+0000 2024-08-19T15:41:22.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:41:22.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:41:22.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:41:22.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:41:22.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:41:22.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:41:22.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:41:22.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:41:22.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:41:02.137038+0000 '>' 2024-08-19T15:41:02.137038+0000 2024-08-19T15:41:22.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:41:23.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:41:23.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:41:23.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:41:23.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:41:23.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:41:23.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:41:23.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:41:24.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:41:02.137038+0000 '>' 2024-08-19T15:41:02.137038+0000 2024-08-19T15:41:24.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:41:25.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:41:25.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:41:25.225 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:41:25.225 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:41:25.225 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:41:25.225 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:41:25.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:41:25.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:41:02.137038+0000 '>' 2024-08-19T15:41:02.137038+0000 2024-08-19T15:41:25.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:41:26.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:41:26.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:41:26.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:41:26.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:41:26.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:41:26.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:41:26.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:41:26.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:41:02.137038+0000 '>' 2024-08-19T15:41:02.137038+0000 2024-08-19T15:41:26.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:41:27.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:41:27.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:41:27.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:41:27.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:41:27.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:41:27.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:41:27.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:41:28.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:41:23.062652+0000 '>' 2024-08-19T15:41:02.137038+0000 2024-08-19T15:41:28.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:41:28.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: seq 1 60 2024-08-19T15:41:28.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: for f in `seq 1 60` 2024-08-19T15:41:28.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: ceph -s 2024-08-19T15:41:28.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: grep '1/1 objects unfound' 2024-08-19T15:41:28.363 INFO:tasks.workunit.client.0.smithi102.stdout: 1/1 objects unfound (100.000%) 2024-08-19T15:41:28.363 INFO:tasks.workunit.client.0.smithi102.stdout: 1/1 objects unfound (100.000%) 2024-08-19T15:41:28.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: break 2024-08-19T15:41:28.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: ceph -s 2024-08-19T15:41:28.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: grep '4 up' 2024-08-19T15:41:28.698 INFO:tasks.workunit.client.0.smithi102.stdout: osd: 4 osds: 4 up (since 12s), 4 in (since 59s) 2024-08-19T15:41:28.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: ceph -s 2024-08-19T15:41:28.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: grep '4 in' 2024-08-19T15:41:29.033 INFO:tasks.workunit.client.0.smithi102.stdout: osd: 4 osds: 4 up (since 12s), 4 in (since 60s) 2024-08-19T15:41:29.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: ceph -s 2024-08-19T15:41:29.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: grep '1/1 objects unfound' 2024-08-19T15:41:29.363 INFO:tasks.workunit.client.0.smithi102.stdout: 1/1 objects unfound (100.000%) 2024-08-19T15:41:29.363 INFO:tasks.workunit.client.0.smithi102.stdout: 1/1 objects unfound (100.000%) 2024-08-19T15:41:29.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T15:41:29.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:41:29.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:41:29.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:41:29.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:41:29.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:41:29.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:41:29.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:41:29.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:41:29.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:41:29.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:41:29.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:41:29.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:41:29.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:41:29.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:41:29.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:41:29.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:41:29.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:41:29.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:41:29.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:41:29.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:41:29.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:41:29.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:41:29.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:41:29.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:41:29.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:41:29.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:41:29.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:41:29.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:41:29.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-08-19T15:41:29.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-08-19T15:41:29.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-08-19T15:41:29.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-08-19T15:41:29.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:41:29.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:41:29.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:41:29.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:41:29.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:41:29.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:41:29.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:41:29.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:41:29.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:41:29.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:41:29.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:41:29.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:41:29.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:41:29.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:41:29.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:41:29.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:41:29.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:41:29.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:41:29.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:41:29.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:41:29.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:41:29.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:41:29.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:41:29.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:41:29.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:41:29.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:41:29.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:41:29.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:41:29.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-08-19T15:41:29.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T15:41:29.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:41:29.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:41:29.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.57280 2024-08-19T15:41:29.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T15:41:29.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T15:41:29.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T15:41:29.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-08-19T15:41:29.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_unfound_erasure_coded_overwrites td/osd-scrub-repair 2024-08-19T15:41:29.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:907: TEST_unfound_erasure_coded_overwrites: '[' true = true ']' 2024-08-19T15:41:29.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:908: TEST_unfound_erasure_coded_overwrites: unfound_erasure_coded td/osd-scrub-repair true 2024-08-19T15:41:29.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: unfound_erasure_coded: local dir=td/osd-scrub-repair 2024-08-19T15:41:29.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:853: unfound_erasure_coded: local allow_overwrites=true 2024-08-19T15:41:29.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:854: unfound_erasure_coded: local poolname=ecpool 2024-08-19T15:41:29.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:855: unfound_erasure_coded: local payload=ABCDEF 2024-08-19T15:41:29.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2024-08-19T15:41:29.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-08-19T15:41:29.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T15:41:29.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T15:41:29.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T15:41:29.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-08-19T15:41:29.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-08-19T15:41:29.889 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T15:41:29.889 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:41:29.889 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:41:29.890 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:41:29.890 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:41:29.890 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:41:29.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:41:29.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T15:41:30.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T15:41:30.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T15:41:30.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:41:30.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:41:30.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T15:41:30.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T15:41:30.083 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:41:30.084 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:41:30.084 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:41:30.084 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:41:30.084 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:41:30.084 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:41:30.085 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:41:30.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:41:30.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get fsid 2024-08-19T15:41:30.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T15:41:30.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:41:30.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:41:30.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T15:41:30.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T15:41:30.163 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:41:30.163 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:41:30.163 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:41:30.163 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:41:30.163 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:41:30.163 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:41:30.163 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.57280/ceph-mon.a.asok 2024-08-19T15:41:30.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:41:30.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.57280/ceph-mon.a.asok config get mon_host 2024-08-19T15:41:30.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2024-08-19T15:41:30.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-08-19T15:41:30.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T15:41:30.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T15:41:30.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T15:41:30.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-08-19T15:41:30.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T15:41:31.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T15:41:31.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:41:31.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:41:31.051 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:41:31.051 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:41:31.051 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:41:31.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:41:31.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:41:31.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:41:31.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: seq 0 3 2024-08-19T15:41:31.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-08-19T15:41:31.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2024-08-19T15:41:31.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:41:31.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:41:31.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T15:41:31.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:41:31.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:41:31.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:41:31.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:41:31.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:41:31.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:41:31.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:41:31.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:41:31.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:41:31.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:41:31.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:41:31.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:41:31.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:41:31.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:41:31.087 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:41:31.087 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:41:31.087 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:41:31.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:41:31.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:41:31.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:41:31.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:41:31.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:41:31.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:41:31.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:41:31.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:41:31.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:41:31.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:41:31.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:41:31.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:41:31.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:41:31.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:41:31.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:41:31.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=427d456e-d733-4da3-acbe-9238feeb9475 2024-08-19T15:41:31.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 427d456e-d733-4da3-acbe-9238feeb9475' 2024-08-19T15:41:31.089 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 427d456e-d733-4da3-acbe-9238feeb9475 2024-08-19T15:41:31.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:41:31.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCrZ8NmehAkBhAAKkwsBWEvPdDdFUuW2JfqZg== 2024-08-19T15:41:31.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCrZ8NmehAkBhAAKkwsBWEvPdDdFUuW2JfqZg=="}' 2024-08-19T15:41:31.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 427d456e-d733-4da3-acbe-9238feeb9475 -i td/osd-scrub-repair/0/new.json 2024-08-19T15:41:31.306 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:41:31.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-08-19T15:41:31.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCrZ8NmehAkBhAAKkwsBWEvPdDdFUuW2JfqZg== --osd-uuid 427d456e-d733-4da3-acbe-9238feeb9475 2024-08-19T15:41:31.345 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:41:31.344+0000 7f1248824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:41:31.357 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:41:31.356+0000 7f1248824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:41:31.358 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:41:31.357+0000 7f1248824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:41:31.359 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:41:31.358+0000 7f1248824640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:41:31.359 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:41:31.358+0000 7f1248824640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:41:31.359 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:41:31.358+0000 7f1248824640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-08-19T15:41:31.449 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:41:31.448+0000 7f1248824640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-08-19T15:41:36.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-08-19T15:41:36.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:41:36.808 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T15:41:36.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T15:41:36.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:41:37.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T15:41:37.165 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:41:37.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:41:37.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:41:37.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:41:37.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:41:37.192 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:41:37.190+0000 7f991523a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:41:37.198 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:41:37.197+0000 7f991523a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:41:37.199 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:41:37.198+0000 7f991523a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:41:37.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T15:41:37.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:41:37.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:41:37.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:41:37.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:41:37.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:37.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:41:37.435 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:41:37.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:37.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:37.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:37.729 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:41:37.728+0000 7f991523a640 -1 Falling back to public interface 2024-08-19T15:41:38.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:38.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:38.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:41:38.703 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:41:38.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:38.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:38.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:39.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:39.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:39.979 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:41:39.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:41:39.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:39.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:40.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:41:40.015+0000 7f991523a640 -1 osd.0 0 log_to_monitors true 2024-08-19T15:41:40.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:41.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:41.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:41.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:41:41.268 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:41:41.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:41.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:41.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:42.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:42.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:42.543 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:41:42.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:41:42.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:42.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:42.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:43.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:43.929 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:41:43.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:43.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:41:43.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:43.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:44.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:45.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:45.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:45.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:41:45.214 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:41:45.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:45.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:45.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:46.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:46.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:46.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:41:46.514 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:41:46.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:46.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:46.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:47.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:47.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:47.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:41:47.911 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:41:47.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:47.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:48.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:49.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:49.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:49.188 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:41:49.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:41:49.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:49.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:49.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:50.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:50.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:50.460 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:41:50.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:41:50.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:50.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:50.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:51.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:51.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:51.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:41:51.743 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:41:51.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:51.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:52.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:53.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:53.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:53.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:41:53.269 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:41:53.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:53.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:53.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:54.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:54.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:54.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:41:54.932 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:41:54.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:54.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:55.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:56.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:56.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:56.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:41:56.683 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:41:56.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:56.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:57.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:58.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:58.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:58.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:41:58.111 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:41:58.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:58.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:58.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:41:59.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:41:59.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:41:59.540 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:41:59.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:41:59.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:41:59.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:41:59.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:00.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:00.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:00.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:42:00.923 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:42:00.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:00.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:42:01.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:02.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:02.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:02.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:42:02.351 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:42:02.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:02.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:42:02.621 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/653550583,v1:127.0.0.1:6803/653550583] [v2:127.0.0.1:6804/653550583,v1:127.0.0.1:6805/653550583] exists,up 427d456e-d733-4da3-acbe-9238feeb9475 2024-08-19T15:42:02.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:42:02.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:42:02.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:42:02.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-08-19T15:42:02.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2024-08-19T15:42:02.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:42:02.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:42:02.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T15:42:02.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:42:02.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-08-19T15:42:02.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:42:02.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:42:02.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:42:02.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:42:02.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-08-19T15:42:02.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-08-19T15:42:02.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:42:02.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:42:02.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:42:02.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:42:02.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:42:02.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:42:02.628 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:42:02.628 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:42:02.628 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:42:02.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:42:02.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:42:02.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:42:02.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:42:02.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:42:02.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:42:02.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:42:02.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:42:02.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:42:02.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:42:02.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:42:02.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:42:02.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:42:02.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-08-19T15:42:02.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:42:02.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=43fe0dcc-fa99-41f3-9295-94042771ceda 2024-08-19T15:42:02.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 43fe0dcc-fa99-41f3-9295-94042771ceda' 2024-08-19T15:42:02.631 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 43fe0dcc-fa99-41f3-9295-94042771ceda 2024-08-19T15:42:02.631 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:42:02.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDKZ8Nm4vhTJhAAjMIm2+H9u4iq8kKngJfX9w== 2024-08-19T15:42:02.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDKZ8Nm4vhTJhAAjMIm2+H9u4iq8kKngJfX9w=="}' 2024-08-19T15:42:02.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 43fe0dcc-fa99-41f3-9295-94042771ceda -i td/osd-scrub-repair/1/new.json 2024-08-19T15:42:02.968 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:42:02.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-08-19T15:42:02.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKZ8Nm4vhTJhAAjMIm2+H9u4iq8kKngJfX9w== --osd-uuid 43fe0dcc-fa99-41f3-9295-94042771ceda 2024-08-19T15:42:03.009 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:03.008+0000 7f5923418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:42:03.044 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:03.043+0000 7f5923418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:42:03.045 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:03.044+0000 7f5923418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:42:03.046 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:03.045+0000 7f5923418640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:42:03.046 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:03.045+0000 7f5923418640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:42:03.046 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:03.045+0000 7f5923418640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-08-19T15:42:03.119 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:03.118+0000 7f5923418640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-08-19T15:42:08.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-08-19T15:42:08.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:42:08.601 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T15:42:08.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T15:42:08.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:42:08.951 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:42:08.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T15:42:08.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:42:08.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:42:08.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:42:08.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:42:08.978 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:08.977+0000 7f3982812640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:42:08.980 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:08.979+0000 7f3982812640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:42:08.981 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:08.980+0000 7f3982812640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:42:09.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T15:42:09.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:42:09.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:42:09.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:42:09.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:42:09.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:09.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:42:09.239 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:42:09.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:09.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:42:09.506 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:09.505+0000 7f3982812640 -1 Falling back to public interface 2024-08-19T15:42:09.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:10.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:10.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:10.514 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:42:10.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:42:10.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:10.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:42:10.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:11.784 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:11.782+0000 7f3982812640 -1 osd.1 0 log_to_monitors true 2024-08-19T15:42:11.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:11.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:11.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:42:11.788 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:42:11.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:11.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:42:12.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:13.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:13.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:13.094 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:42:13.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:42:13.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:13.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:42:13.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:14.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:14.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:14.461 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:42:14.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:42:14.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:14.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:42:14.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:15.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:15.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:15.791 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:42:15.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:42:15.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:15.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:42:16.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:17.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:17.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:17.093 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:42:17.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:42:17.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:17.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:42:17.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:18.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:18.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:18.480 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:42:18.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:42:18.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:18.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:42:18.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:19.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:19.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:19.758 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:42:19.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:42:19.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:19.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:42:20.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:21.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:21.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:21.061 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:42:21.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:42:21.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:21.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:42:21.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:22.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:22.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:22.434 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:42:22.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:42:22.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:22.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:42:22.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:23.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:23.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:23.749 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:42:23.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:42:23.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:23.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:42:24.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:25.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:25.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:25.030 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:42:25.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:42:25.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:25.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:42:26.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:27.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:27.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:27.004 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:42:27.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:42:27.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:27.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:42:27.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:28.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:28.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:28.423 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:42:28.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:42:28.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:28.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:42:29.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:30.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:30.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:30.015 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:42:30.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:42:30.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:30.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:42:30.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:31.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:31.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:31.303 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:42:31.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:42:31.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:31.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:42:32.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:33.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:33.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:33.006 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:42:33.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:42:33.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:33.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:42:33.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:34.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:34.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:34.439 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:42:34.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:42:34.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:34.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:42:34.802 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3341298447,v1:127.0.0.1:6811/3341298447] [v2:127.0.0.1:6812/3341298447,v1:127.0.0.1:6813/3341298447] exists,up 43fe0dcc-fa99-41f3-9295-94042771ceda 2024-08-19T15:42:34.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:42:34.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:42:34.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:42:34.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-08-19T15:42:34.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2024-08-19T15:42:34.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:42:34.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:42:34.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T15:42:34.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:42:34.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:42:34.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:42:34.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:42:34.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:42:34.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:42:34.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T15:42:34.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T15:42:34.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:42:34.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:42:34.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:42:34.810 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:42:34.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:42:34.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:42:34.811 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:42:34.811 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:42:34.811 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:42:34.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:42:34.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:42:34.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:42:34.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:42:34.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:42:34.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:42:34.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:42:34.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:42:34.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:42:34.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:42:34.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:42:34.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:42:34.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:42:34.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T15:42:34.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:42:34.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f474438e-9657-46cc-b7d5-e9dfebeba7e2 2024-08-19T15:42:34.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f474438e-9657-46cc-b7d5-e9dfebeba7e2' 2024-08-19T15:42:34.812 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 f474438e-9657-46cc-b7d5-e9dfebeba7e2 2024-08-19T15:42:34.813 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:42:34.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDqZ8Nm75IWMRAArK5D5jz8nVW3R+t/SL129A== 2024-08-19T15:42:34.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDqZ8Nm75IWMRAArK5D5jz8nVW3R+t/SL129A=="}' 2024-08-19T15:42:34.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f474438e-9657-46cc-b7d5-e9dfebeba7e2 -i td/osd-scrub-repair/2/new.json 2024-08-19T15:42:35.143 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:42:35.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-08-19T15:42:35.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDqZ8Nm75IWMRAArK5D5jz8nVW3R+t/SL129A== --osd-uuid f474438e-9657-46cc-b7d5-e9dfebeba7e2 2024-08-19T15:42:35.183 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:35.182+0000 7f9072412640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:42:35.227 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:35.225+0000 7f9072412640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:42:35.227 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:35.227+0000 7f9072412640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:42:35.228 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:35.227+0000 7f9072412640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:42:35.228 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:35.227+0000 7f9072412640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:42:35.228 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:35.227+0000 7f9072412640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-08-19T15:42:35.294 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:35.292+0000 7f9072412640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-08-19T15:42:41.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-08-19T15:42:41.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:42:41.230 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T15:42:41.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T15:42:41.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:42:41.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T15:42:41.577 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T15:42:41.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:42:41.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:42:41.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:42:41.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:42:41.604 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:41.603+0000 7f2ec5e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:42:41.606 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:41.605+0000 7f2ec5e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:42:41.607 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:41.606+0000 7f2ec5e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:42:41.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T15:42:41.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:42:41.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T15:42:41.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:42:41.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:42:41.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:41.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:42:41.844 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:42:41.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:41.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:42:42.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:42.139 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:42.138+0000 7f2ec5e0d640 -1 Falling back to public interface 2024-08-19T15:42:43.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:43.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:43.111 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:42:43.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:42:43.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:43.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:42:43.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:44.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:44.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:44.411 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:42:44.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:42:44.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:44.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:42:44.468 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:42:44.467+0000 7f2ec5e0d640 -1 osd.2 0 log_to_monitors true 2024-08-19T15:42:44.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:45.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:45.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:45.681 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:42:45.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:42:45.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:45.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:42:45.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:46.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:46.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:46.962 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:42:46.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:42:46.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:46.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:42:47.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:48.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:48.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:48.340 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:42:48.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:42:48.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:48.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:42:48.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:49.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:49.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:49.619 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:42:49.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:42:49.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:49.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:42:49.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:50.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:50.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:50.893 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:42:50.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:42:50.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:50.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:42:51.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:52.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:52.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:52.306 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:42:52.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:42:52.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:52.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:42:52.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:53.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:53.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:53.579 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:42:53.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:42:53.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:53.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:42:53.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:54.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:54.876 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:42:54.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:54.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:42:54.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:54.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:42:55.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:56.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:56.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:56.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:42:56.314 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:42:56.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:56.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:42:56.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:57.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:57.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:57.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:42:57.589 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:42:57.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:57.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:42:58.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:42:59.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:42:59.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:42:59.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:42:59.539 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:42:59.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:42:59.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:42:59.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:00.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:00.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:00.966 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:43:00.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:43:00.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:00.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:43:01.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:02.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:02.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:02.387 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:43:02.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:43:02.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:02.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:43:03.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:04.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:04.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:04.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:43:04.098 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:43:04.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:04.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:43:04.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:05.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:05.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:05.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:43:05.695 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:43:05.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:05.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:43:06.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:07.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:07.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:07.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:43:07.545 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:43:07.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:07.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:43:07.821 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1420022704,v1:127.0.0.1:6819/1420022704] [v2:127.0.0.1:6820/1420022704,v1:127.0.0.1:6821/1420022704] exists,up f474438e-9657-46cc-b7d5-e9dfebeba7e2 2024-08-19T15:43:07.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:43:07.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:43:07.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:43:07.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-08-19T15:43:07.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2024-08-19T15:43:07.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-08-19T15:43:07.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:43:07.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-08-19T15:43:07.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:43:07.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T15:43:07.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:43:07.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:43:07.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:43:07.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:43:07.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T15:43:07.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T15:43:07.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:43:07.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:43:07.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:43:07.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:43:07.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:43:07.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:43:07.827 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:43:07.827 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:43:07.827 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:43:07.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:43:07.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:43:07.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:43:07.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:43:07.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:43:07.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:43:07.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:43:07.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:43:07.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:43:07.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:43:07.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:43:07.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:43:07.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:43:07.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T15:43:07.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:43:07.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0d264c2b-584c-412a-8448-90a9e380a0ef 2024-08-19T15:43:07.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 0d264c2b-584c-412a-8448-90a9e380a0ef' 2024-08-19T15:43:07.830 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 0d264c2b-584c-412a-8448-90a9e380a0ef 2024-08-19T15:43:07.830 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:43:07.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALaMNmBkA5MhAA1lYWV/BljuKZLTjt3EZfwQ== 2024-08-19T15:43:07.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALaMNmBkA5MhAA1lYWV/BljuKZLTjt3EZfwQ=="}' 2024-08-19T15:43:07.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0d264c2b-584c-412a-8448-90a9e380a0ef -i td/osd-scrub-repair/3/new.json 2024-08-19T15:43:08.246 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:43:08.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-08-19T15:43:08.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALaMNmBkA5MhAA1lYWV/BljuKZLTjt3EZfwQ== --osd-uuid 0d264c2b-584c-412a-8448-90a9e380a0ef 2024-08-19T15:43:08.287 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:43:08.286+0000 7fd44e157640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:43:08.331 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:43:08.330+0000 7fd44e157640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:43:08.332 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:43:08.331+0000 7fd44e157640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:43:08.333 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:43:08.331+0000 7fd44e157640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-08-19T15:43:08.333 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:43:08.331+0000 7fd44e157640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-08-19T15:43:08.333 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:43:08.331+0000 7fd44e157640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-08-19T15:43:08.406 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:43:08.405+0000 7fd44e157640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-08-19T15:43:14.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-08-19T15:43:14.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:43:14.474 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-08-19T15:43:14.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-08-19T15:43:14.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:43:14.817 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-08-19T15:43:14.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-08-19T15:43:14.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:43:14.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:43:14.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:43:14.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:43:14.843 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:43:14.842+0000 7fd281dd6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:43:14.846 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:43:14.845+0000 7fd281dd6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:43:14.847 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:43:14.846+0000 7fd281dd6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:43:15.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-08-19T15:43:15.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:43:15.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T15:43:15.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:43:15.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:43:15.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:15.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:43:15.091 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:43:15.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:15.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:15.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:15.380 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:43:15.379+0000 7fd281dd6640 -1 Falling back to public interface 2024-08-19T15:43:16.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:16.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:16.356 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:43:16.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:43:16.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:16.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:16.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:17.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:17.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:17.628 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:43:17.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:43:17.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:17.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:17.665 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:43:17.664+0000 7fd281dd6640 -1 osd.3 0 log_to_monitors true 2024-08-19T15:43:17.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:18.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:18.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:18.975 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:43:18.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:43:18.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:18.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:19.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:20.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:20.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:20.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:43:20.309 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:43:20.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:20.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:20.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:21.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:21.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:21.687 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:43:21.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:43:21.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:21.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:21.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:22.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:22.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:22.964 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:43:22.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:43:22.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:22.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:23.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:24.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:24.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:24.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:43:24.290 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:43:24.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:24.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:24.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:25.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:25.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:25.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:43:25.659 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:43:25.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:25.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:25.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:26.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:26.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:26.966 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:43:26.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:43:26.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:26.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:27.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:28.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:28.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:28.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:43:28.293 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:43:28.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:28.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:28.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:29.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:29.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:29.701 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:43:29.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:43:29.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:29.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:29.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:30.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:30.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:30.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:43:30.977 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:43:30.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:30.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:31.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:32.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:32.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:32.611 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:43:32.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:43:32.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:32.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:33.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:34.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:34.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:34.314 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:43:34.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:43:34.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:34.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:34.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:35.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:35.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:35.927 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:43:35.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:43:35.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:35.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:36.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:37.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:37.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:37.707 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:43:37.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:43:37.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:37.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:38.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:39.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:39.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:39.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:43:39.327 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:43:39.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:39.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:39.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:40.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:40.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:40.724 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:43:40.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:43:40.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:40.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:41.011 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2457754289,v1:127.0.0.1:6827/2457754289] [v2:127.0.0.1:6828/2457754289,v1:127.0.0.1:6829/2457754289] exists,up 0d264c2b-584c-412a-8448-90a9e380a0ef 2024-08-19T15:43:41.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:43:41.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:43:41.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:43:41.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: unfound_erasure_coded: create_ec_pool ecpool true k=2 m=2 2024-08-19T15:43:41.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-08-19T15:43:41.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-08-19T15:43:41.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-08-19T15:43:41.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-08-19T15:43:41.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-08-19T15:43:41.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-08-19T15:43:41.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-08-19T15:43:41.996 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'ecpool' created 2024-08-19T15:43:42.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:43:43.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-08-19T15:43:43.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-08-19T15:43:43.377 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 allow_ec_overwrites to true 2024-08-19T15:43:43.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-08-19T15:43:43.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:43:43.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:43:43.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:43:43.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:43:43.389 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:43:43.389 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:43:43.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:43:43.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:43:43.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:43:43.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:43:43.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:43:43.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:43:43.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:43:43.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:43:43.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:43:43.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:43:43.867 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:43:43.867 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:43:43.868 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-08-19T15:43:43.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:43:43.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:43:43.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:43:43.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-08-19T15:43:43.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-08-19T15:43:43.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-08-19T15:43:43.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:43:43.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:43:44.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=38654705687 2024-08-19T15:43:44.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 38654705687 2024-08-19T15:43:44.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-38654705687' 2024-08-19T15:43:44.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:43:44.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:43:44.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542160 2024-08-19T15:43:44.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542160 2024-08-19T15:43:44.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-38654705687 2-60129542160' 2024-08-19T15:43:44.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:43:44.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:43:44.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378632 2024-08-19T15:43:44.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378632 2024-08-19T15:43:44.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-38654705687 2-60129542160 3-81604378632' 2024-08-19T15:43:44.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:43:44.420 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-08-19T15:43:44.420 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:43:44.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:43:44.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-08-19T15:43:44.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:43:44.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-08-19T15:43:44.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-08-19T15:43:44.423 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836505 2024-08-19T15:43:44.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:43:44.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-08-19T15:43:44.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:43:45.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:43:45.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:43:45.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-08-19T15:43:45.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:43:45.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-38654705687 2024-08-19T15:43:45.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:43:45.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:43:45.962 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-38654705687 2024-08-19T15:43:45.962 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:43:45.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=38654705687 2024-08-19T15:43:45.963 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 38654705687 2024-08-19T15:43:45.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 38654705687' 2024-08-19T15:43:45.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:43:46.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 38654705687 -lt 38654705687 2024-08-19T15:43:46.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:43:46.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542160 2024-08-19T15:43:46.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:43:46.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:43:46.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542160 2024-08-19T15:43:46.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:43:46.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542160 2024-08-19T15:43:46.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542160' 2024-08-19T15:43:46.235 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 60129542160 2024-08-19T15:43:46.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:43:46.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542160 -lt 60129542160 2024-08-19T15:43:46.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:43:46.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-81604378632 2024-08-19T15:43:46.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:43:46.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:43:46.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-81604378632 2024-08-19T15:43:46.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:43:46.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378632 2024-08-19T15:43:46.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 81604378632' 2024-08-19T15:43:46.505 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 81604378632 2024-08-19T15:43:46.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:43:46.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378632 -lt 81604378632 2024-08-19T15:43:46.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:43:46.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:43:46.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:43:47.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:43:47.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:43:47.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:43:47.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:43:47.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:43:47.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:43:47.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:43:47.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:43:47.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:43:47.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:43:47.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:43:47.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:43:47.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:43:47.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:43:47.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:43:47.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-08-19T15:43:47.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-08-19T15:43:47.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-08-19T15:43:47.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-08-19T15:43:47.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-08-19T15:43:47.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-08-19T15:43:47.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-08-19T15:43:47.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-08-19T15:43:48.041 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:43:48.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-08-19T15:43:48.552 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:43:48.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-08-19T15:43:48.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-08-19T15:43:48.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-08-19T15:43:48.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: get_primary ecpool SOMETHING 2024-08-19T15:43:48.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-08-19T15:43:48.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-08-19T15:43:48.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-08-19T15:43:48.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:43:49.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: local primary=1 2024-08-19T15:43:49.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: get_osds ecpool SOMETHING 2024-08-19T15:43:49.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-08-19T15:43:49.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: sed -e s/1// 2024-08-19T15:43:49.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-08-19T15:43:49.144 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-08-19T15:43:49.144 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-19T15:43:49.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-08-19T15:43:49.414 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-08-19T15:43:49.414 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-08-19T15:43:49.414 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-08-19T15:43:49.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 3 2024-08-19T15:43:49.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: osds=('0' '2' '3') 2024-08-19T15:43:49.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: local -a osds 2024-08-19T15:43:49.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: local not_primary_first=0 2024-08-19T15:43:49.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: unfound_erasure_coded: local not_primary_second=2 2024-08-19T15:43:49.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:871: unfound_erasure_coded: local not_primary_third=3 2024-08-19T15:43:49.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:876: unfound_erasure_coded: pids= 2024-08-19T15:43:49.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T15:43:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T15:43:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T15:43:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 466052"' 2024-08-19T15:43:49.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 466052' 2024-08-19T15:43:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:878: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-08-19T15:43:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T15:43:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T15:43:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 466053"' 2024-08-19T15:43:49.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 466053' 2024-08-19T15:43:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:879: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T15:43:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-08-19T15:43:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T15:43:49.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-08-19T15:43:49.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/466055: /' 2024-08-19T15:43:49.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 466056"' 2024-08-19T15:43:49.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 466056' 2024-08-19T15:43:49.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:880: unfound_erasure_coded: wait_background pids 2024-08-19T15:43:49.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-08-19T15:43:49.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 466052 466053 466056' 2024-08-19T15:43:49.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-08-19T15:43:49.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T15:43:49.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 466052 2024-08-19T15:43:49.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/466058: /' 2024-08-19T15:43:49.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T15:43:49.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/466061: /' 2024-08-19T15:43:52.142 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:43:52.143 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:43:52.143 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-08-19T15:43:52.143 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:43:52.143 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-08-19T15:43:52.143 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:43:52.143 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:43:52.143 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-08-19T15:43:52.143 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:43:52.143 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-08-19T15:43:52.143 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:43:52.143 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:43:52.143 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:43:52.144 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:43:52.144 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:43:52.144 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:43:52.144 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-08-19T15:43:52.144 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:43:52.144 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:43:52.144 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-08-19T15:43:52.144 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:43:52.144 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:43:52.144 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-08-19T15:43:52.144 INFO:tasks.workunit.client.0.smithi102.stderr:466058: remove 2#1:eb822e21:::SOMETHING:head# 2024-08-19T15:43:52.144 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-08-19T15:43:52.144 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:43:52.145 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:43:52.145 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-19T15:43:52.145 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:43:52.145 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-08-19T15:43:52.145 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:43:52.145 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:43:52.146 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:43:52.146 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:43:52.146 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-08-19T15:43:52.146 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-08-19T15:43:52.146 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:43:52.147 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:43:52.147 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:43:52.147 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:43:52.147 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:43:52.147 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:43:52.147 INFO:tasks.workunit.client.0.smithi102.stderr:466058: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:43:52.147 INFO:tasks.workunit.client.0.smithi102.stderr:466058: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:43:52.147 INFO:tasks.workunit.client.0.smithi102.stderr:466058: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:43:52.147 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:43:52.147 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:43:52.147 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:43:52.147 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:43:52.147 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:43:52.148 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:43:52.148 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:43:52.148 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:43:52.148 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:43:52.148 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:43:52.148 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:43:52.148 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-08-19T15:43:52.148 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-19T15:43:52.148 INFO:tasks.workunit.client.0.smithi102.stderr:466058: start osd.2 2024-08-19T15:43:52.148 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:43:52.430 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:43:52.430 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:43:52.431 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T15:43:52.431 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:43:52.431 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T15:43:52.431 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:43:52.431 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:43:52.431 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T15:43:52.431 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:43:52.431 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-08-19T15:43:52.431 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:43:52.431 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:43:52.431 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:43:52.431 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:43:52.431 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:43:52.432 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:43:52.432 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-08-19T15:43:52.432 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:43:52.432 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:43:52.432 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T15:43:52.432 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:43:52.432 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:43:52.432 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-08-19T15:43:52.432 INFO:tasks.workunit.client.0.smithi102.stderr:466055: remove 1#1:eb822e21:::SOMETHING:head# 2024-08-19T15:43:52.432 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-08-19T15:43:52.432 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:43:52.432 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:43:52.432 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:43:52.433 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:43:52.433 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-08-19T15:43:52.433 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:43:52.433 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:43:52.433 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: 466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-08-19T15:43:52.433 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T15:43:52.433 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-08-19T15:43:52.433 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T15:43:52.434 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T15:43:52.434 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-08-19T15:43:52.434 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T15:43:52.434 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-08-19T15:43:52.434 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T15:43:52.434 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-08-19T15:43:52.434 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:43:52.434 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:43:52.434 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:43:52.434 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:43:52.434 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:43:52.434 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:43:52.435 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-08-19T15:43:52.435 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-08-19T15:43:52.435 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T15:43:52.435 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-08-19T15:43:52.435 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T15:43:52.435 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-08-19T15:43:52.435 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-08-19T15:43:52.435 INFO:tasks.workunit.client.0.smithi102.stderr:466061: remove 3#1:eb822e21:::SOMETHING:head# 2024-08-19T15:43:52.435 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-08-19T15:43:52.435 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-08-19T15:43:52.435 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:43:52.435 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-19T15:43:52.435 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:43:52.435 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-08-19T15:43:52.436 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-08-19T15:43:52.436 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:43:52.436 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:43:52.436 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:43:52.436 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-08-19T15:43:52.436 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-08-19T15:43:52.436 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:43:52.437 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:43:52.437 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:43:52.437 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:43:52.437 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:43:52.437 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:43:52.437 INFO:tasks.workunit.client.0.smithi102.stderr:466055: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:43:52.437 INFO:tasks.workunit.client.0.smithi102.stderr:466055: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:43:52.437 INFO:tasks.workunit.client.0.smithi102.stderr:466055: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:43:52.437 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:43:52.437 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:43:52.437 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:43:52.438 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:43:52.438 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:43:52.438 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:43:52.438 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:43:52.438 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:43:52.438 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:43:52.438 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:43:52.438 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:43:52.438 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-08-19T15:43:52.438 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:43:52.438 INFO:tasks.workunit.client.0.smithi102.stderr:466055: start osd.0 2024-08-19T15:43:52.438 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:43:52.439 INFO:tasks.workunit.client.0.smithi102.stderr:activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:43:52.439 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:43:52.439 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-08-19T15:43:52.439 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-08-19T15:43:52.439 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:43:52.439 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:43:52.439 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-08-19T15:43:52.439 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:43:52.439 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:43:52.439 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:43:52.439 INFO:tasks.workunit.client.0.smithi102.stderr:466061: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:43:52.439 INFO:tasks.workunit.client.0.smithi102.stderr:466061: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:43:52.440 INFO:tasks.workunit.client.0.smithi102.stderr:466061: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:43:52.440 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:43:52.440 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' 2024-08-19T15:43:52.440 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:43:52.440 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-08-19T15:43:52.440 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-08-19T15:43:52.440 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:43:52.440 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:43:52.440 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:43:52.440 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:43:52.440 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:43:52.441 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:43:52.441 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-08-19T15:43:52.441 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-19T15:43:52.441 INFO:tasks.workunit.client.0.smithi102.stderr:466061: start osd.3 2024-08-19T15:43:52.441 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1ef038fb-3dcb-4ed8-83a2-bbfbb8211b5f --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.57280/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:43:56.780 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-08-19T15:43:56.781 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-19T15:43:56.781 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:43:56.781 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:43:56.781 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:43:56.782 INFO:tasks.workunit.client.0.smithi102.stderr:466058: 2024-08-19T15:43:52.173+0000 7f0581333640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:43:56.782 INFO:tasks.workunit.client.0.smithi102.stderr:466058: 2024-08-19T15:43:52.176+0000 7f0581333640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:43:56.782 INFO:tasks.workunit.client.0.smithi102.stderr:466058: 2024-08-19T15:43:52.177+0000 7f0581333640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:43:56.782 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-19T15:43:56.782 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:43:56.783 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T15:43:56.783 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:43:56.783 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:43:56.783 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:56.783 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:43:56.783 INFO:tasks.workunit.client.0.smithi102.stderr:466058: 0 2024-08-19T15:43:56.784 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:56.784 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:43:56.784 INFO:tasks.workunit.client.0.smithi102.stderr:466058: 2024-08-19T15:43:52.692+0000 7f0581333640 -1 Falling back to public interface 2024-08-19T15:43:56.784 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:56.784 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:56.784 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:56.785 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:43:56.785 INFO:tasks.workunit.client.0.smithi102.stderr:466058: 1 2024-08-19T15:43:56.785 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:56.785 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:43:56.785 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:56.785 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:56.786 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:56.786 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:43:56.786 INFO:tasks.workunit.client.0.smithi102.stderr:466058: 2 2024-08-19T15:43:56.787 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:56.787 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:43:56.787 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:56.788 INFO:tasks.workunit.client.0.smithi102.stderr:466058: 2024-08-19T15:43:55.423+0000 7f0581333640 -1 osd.2 28 log_to_monitors true 2024-08-19T15:43:56.788 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:56.788 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:56.789 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:43:56.789 INFO:tasks.workunit.client.0.smithi102.stderr:466058: 3 2024-08-19T15:43:56.789 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:56.789 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:43:57.060 INFO:tasks.workunit.client.0.smithi102.stderr:466058: osd.2 up in weight 1 up_from 32 up_thru 0 down_at 29 last_c466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-08-19T15:43:57.060 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:43:57.060 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:43:57.061 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:43:57.061 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:43:57.061 INFO:tasks.workunit.client.0.smithi102.stderr:466055: 2024-08-19T15:43:52.476+0000 7fae5280d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:43:57.061 INFO:tasks.workunit.client.0.smithi102.stderr:466055: 2024-08-19T15:43:52.553+0000 7fae5280d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:43:57.061 INFO:tasks.workunit.client.0.smithi102.stderr:466055: 2024-08-19T15:43:52.554+0000 7fae5280d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:43:57.062 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:43:57.062 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:43:57.062 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:43:57.062 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:43:57.062 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:43:57.062 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:57.063 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:43:57.063 INFO:tasks.workunit.client.0.smithi102.stderr:466055: 0 2024-08-19T15:43:57.063 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:57.063 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:43:57.063 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:57.063 INFO:tasks.workunit.client.0.smithi102.stderr:466055: 2024-08-19T15:43:53.086+0000 7fae5280d640 -1 Falling back to public interface 2024-08-19T15:43:57.064 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:57.064 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:57.064 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:43:57.064 INFO:tasks.workunit.client.0.smithi102.stderr:466055: 1 2024-08-19T15:43:57.064 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:57.065 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:43:57.065 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:57.065 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:57.065 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:57.065 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:43:57.065 INFO:tasks.workunit.client.0.smithi102.stderr:466055: 2 2024-08-19T15:43:57.066 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:57.066 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:43:57.066 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:57.066 INFO:tasks.workunit.client.0.smithi102.stderr:466055: 2024-08-19T15:43:56.043+0000 7fae5280d640 -1 osd.0 28 log_to_monitors true 2024-08-19T15:43:57.066 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:57.066 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:57.067 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:43:57.067 INFO:tasks.workunit.client.0.smithi102.stderr:466055: 3 2024-08-19T15:43:57.067 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:43:57.067 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:57.112 INFO:tasks.workunit.client.0.smithi102.stderr:466055: osd.0 up in weight 1 up_from 34 up_thru 0 down_at 29 last_c466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-08-19T15:43:57.112 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-19T15:43:57.112 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:43:57.112 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:43:57.112 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:43:57.112 INFO:tasks.workunit.client.0.smithi102.stderr:466061: 2024-08-19T15:43:52.461+0000 7f0fd9c32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:43:57.112 INFO:tasks.workunit.client.0.smithi102.stderr:466061: 2024-08-19T15:43:52.552+0000 7f0fd9c32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:43:57.112 INFO:tasks.workunit.client.0.smithi102.stderr:466061: 2024-08-19T15:43:52.554+0000 7f0fd9c32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:43:57.113 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-19T15:43:57.113 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:43:57.113 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T15:43:57.113 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:43:57.113 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:43:57.113 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:57.113 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:43:57.113 INFO:tasks.workunit.client.0.smithi102.stderr:466061: 0 2024-08-19T15:43:57.113 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:57.113 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:57.113 INFO:tasks.workunit.client.0.smithi102.stderr:466061: 2024-08-19T15:43:52.829+0000 7f0fd9c32640 -1 Falling back to public interface 2024-08-19T15:43:57.113 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:57.113 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:57.114 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:57.114 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:43:57.114 INFO:tasks.workunit.client.0.smithi102.stderr:466061: 1 2024-08-19T15:43:57.114 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:57.114 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:57.114 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:57.114 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:57.114 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:57.114 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:43:57.114 INFO:tasks.workunit.client.0.smithi102.stderr:466061: 2 2024-08-19T15:43:57.114 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:57.115 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:57.115 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:43:57.115 INFO:tasks.workunit.client.0.smithi102.stderr:466061: 2024-08-19T15:43:55.748+0000 7f0fd9c32640 -1 osd.3 28 log_to_monitors true 2024-08-19T15:43:57.115 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:43:57.115 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:43:57.115 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:43:57.115 INFO:tasks.workunit.client.0.smithi102.stderr:466061: 3 2024-08-19T15:43:57.115 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:43:57.115 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T15:43:57.639 INFO:tasks.workunit.client.0.smithi102.stderr:466061: osd.3 up in weight 1 up_from 33 up_thru 0 down_at 29 last_clean_interval [14,28) [v2:127.0.0.1:6802/255478027,v1:127.0.0.1:6803/255478027] [v2:127.0.0.1:6804/255478027,v1:127.0.0.1:6805/255478027] exists,up f474438e-9657-46cc-b7d5-e9dfebeba7e2 2024-08-19T15:43:57.639 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:43:57.639 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:43:57.639 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:43:57.640 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:43:57.640 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:43:57.640 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:43:57.640 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:43:57.640 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:43:57.640 INFO:tasks.workunit.client.0.smithi102.stderr:466058: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:43:57.640 INFO:tasks.workunit.client.0.smithi102.stderr:466058: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:43:57.641 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:43:57.641 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:43:57.641 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:43:57.641 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:43:57.641 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:43:57.641 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:43:57.641 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:43:57.642 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:43:57.642 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:43:57.642 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:43:57.642 INFO:tasks.workunit.client.0.smithi102.stderr:466058: 1 2024-08-19T15:43:57.642 INFO:tasks.workunit.client.0.smithi102.stderr:466058: 2 2024-08-19T15:43:57.642 INFO:tasks.workunit.client.0.smithi102.stderr:466058: 3' 2024-08-19T15:43:57.642 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:43:57.642 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:43:57.642 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:43:57.642 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-08-19T15:43:57.642 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-08-19T15:43:57.642 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066' 2024-08-19T15:43:57.642 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:43:57.643 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:43:57.643 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=38654705691 2024-08-19T15:43:57.643 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 38654705691 2024-08-19T15:43:57.643 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-38654705691' 2024-08-19T15:43:57.643 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:43:57.742 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph telllean_interval [19,28) [v2:127.0.0.1:6818/2798510534,v1:127.0.0.1:6819/2798510534] [v2:127.0.0.1:6820/2798510534,v1:127.0.0.1:6821/2798510534] exists,up 0d264c2b-584c-412a-8448-90a9e380a0ef 2024-08-19T15:43:57.742 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:43:57.742 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:43:57.743 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:43:57.743 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:43:57.743 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:43:57.743 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:43:57.743 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:43:57.743 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:43:57.743 INFO:tasks.workunit.client.0.smithi102.stderr:466061: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:43:57.744 INFO:tasks.workunit.client.0.smithi102.stderr:466061: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:43:57.744 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:43:57.744 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:43:57.744 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:43:57.744 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:43:57.744 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:43:57.745 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:43:57.745 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:43:57.745 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:43:57.745 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:43:57.745 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:43:57.745 INFO:tasks.workunit.client.0.smithi102.stderr:466061: 1 2024-08-19T15:43:57.746 INFO:tasks.workunit.client.0.smithi102.stderr:466061: 2 2024-08-19T15:43:57.746 INFO:tasks.workunit.client.0.smithi102.stderr:466061: 3' 2024-08-19T15:43:57.746 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:43:57.746 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:43:57.746 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:43:57.747 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888068 2024-08-19T15:43:57.747 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888068 2024-08-19T15:43:57.747 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888068' 2024-08-19T15:43:57.747 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:43:57.747 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:43:57.747 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=38654705692 2024-08-19T15:43:57.748 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 38654705692 2024-08-19T15:43:57.748 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888068 1-38654705692' 2024-08-19T15:43:57.748 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:43:57.756 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph lean_interval [5,28) [v2:127.0.0.1:6826/3513730281,v1:127.0.0.1:6827/3513730281] [v2:127.0.0.1:6828/3513730281,v1:127.0.0.1:6829/3513730281] exists,up 427d456e-d733-4da3-acbe-9238feeb9475 2024-08-19T15:43:57.756 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:43:57.756 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:43:57.757 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:43:57.757 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T15:43:57.757 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:43:57.757 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:43:57.757 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:43:57.758 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:43:57.758 INFO:tasks.workunit.client.0.smithi102.stderr:466055: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:43:57.758 INFO:tasks.workunit.client.0.smithi102.stderr:466055: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:43:57.758 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:43:57.758 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:43:57.758 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:43:57.759 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:43:57.759 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:43:57.759 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:43:57.759 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:43:57.760 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:43:57.760 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:43:57.760 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:43:57.760 INFO:tasks.workunit.client.0.smithi102.stderr:466055: 1 2024-08-19T15:43:57.760 INFO:tasks.workunit.client.0.smithi102.stderr:466055: 2 2024-08-19T15:43:57.761 INFO:tasks.workunit.client.0.smithi102.stderr:466055: 3' 2024-08-19T15:43:57.761 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:43:57.761 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:43:57.761 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:43:57.761 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888067 2024-08-19T15:43:57.761 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888067 2024-08-19T15:43:57.762 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888067' 2024-08-19T15:43:57.762 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:43:57.762 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:43:57.762 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=38654705693 2024-08-19T15:43:57.762 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 38654705693 2024-08-19T15:43:57.762 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888067 1-38654705693' 2024-08-19T15:43:57.763 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:43:59.803 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph t osd.2 flush_pg_stats 2024-08-19T15:43:59.803 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-08-19T15:43:59.803 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-08-19T15:43:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-38654705691 2-137438953474' 2024-08-19T15:43:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:43:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:43:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920770 2024-08-19T15:43:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920770 2024-08-19T15:43:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-38654705691 2-137438953474 3-141733920770' 2024-08-19T15:43:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:43:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888066 2024-08-19T15:43:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:43:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:43:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888066 2024-08-19T15:43:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:43:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-08-19T15:43:59.805 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2024-08-19T15:43:59.805 INFO:tasks.workunit.client.0.smithi102.stderr:466058: waiting osd.0 seq 146028888066 2024-08-19T15:43:59.805 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:43:59.805 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 146028888066 2024-08-19T15:43:59.805 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:43:59.805 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:43:59.805 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:43:59.805 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888066 2024-08-19T15:43:59.805 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:43:59.805 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-38654705691 2024-08-19T15:43:59.805 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:43:59.805 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:43:59.805 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-38654705691 2024-08-19T15:43:59.805 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:43:59.806 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=38654705691 2024-08-19T15:43:59.806 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 38654705691' 2024-08-19T15:43:59.806 INFO:tasks.workunit.client.0.smithi102.stderr:466058: waiting osd.1 seq 38654705691 2024-08-19T15:43:59.806 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:43:59.806 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 38654705693 -lt 38654705691 2024-08-19T15:43:59.806 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:43:59.841 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubtell osd.2 flush_pg_stats 2024-08-19T15:43:59.841 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953475 2024-08-19T15:43:59.841 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953475 2024-08-19T15:43:59.841 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888068 1-38654705692 2-137438953475' 2024-08-19T15:43:59.841 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:43:59.841 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:43:59.842 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920771 2024-08-19T15:43:59.842 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920771 2024-08-19T15:43:59.842 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888068 1-38654705692 2-137438953475 3-141733920771' 2024-08-19T15:43:59.842 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:43:59.842 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888068 2024-08-19T15:43:59.842 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:43:59.842 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:43:59.842 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888068 2024-08-19T15:43:59.842 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:43:59.842 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888068 2024-08-19T15:43:59.842 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888068' 2024-08-19T15:43:59.842 INFO:tasks.workunit.client.0.smithi102.stderr:466061: waiting osd.0 seq 146028888068 2024-08-19T15:43:59.842 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:43:59.843 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 146028888068 2024-08-19T15:43:59.843 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:43:59.843 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:43:59.843 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:43:59.843 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888068 2024-08-19T15:43:59.843 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:43:59.843 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-38654705692 2024-08-19T15:43:59.843 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:43:59.843 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:43:59.843 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-38654705692 2024-08-19T15:43:59.843 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:43:59.843 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=38654705692 2024-08-19T15:43:59.843 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 38654705692' 2024-08-19T15:43:59.844 INFO:tasks.workunit.client.0.smithi102.stderr:466061: waiting osd.1 seq 38654705692 2024-08-19T15:43:59.844 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:43:59.844 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 38654705693 -lt 38654705692 2024-08-19T15:43:59.844 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:43:59.876 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //homell osd.2 flush_pg_stats 2024-08-19T15:43:59.876 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953476 2024-08-19T15:43:59.877 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953476 2024-08-19T15:43:59.877 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888067 1-38654705693 2-137438953476' 2024-08-19T15:43:59.877 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:43:59.877 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T15:43:59.877 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920772 2024-08-19T15:43:59.877 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920772 2024-08-19T15:43:59.877 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888067 1-38654705693 2-137438953476 3-141733920772' 2024-08-19T15:43:59.877 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:43:59.877 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888067 2024-08-19T15:43:59.877 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:43:59.877 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:43:59.877 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888067 2024-08-19T15:43:59.878 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:43:59.878 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888067 2024-08-19T15:43:59.878 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888067' 2024-08-19T15:43:59.878 INFO:tasks.workunit.client.0.smithi102.stderr:466055: waiting osd.0 seq 146028888067 2024-08-19T15:43:59.878 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:43:59.878 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 146028888067 2024-08-19T15:43:59.878 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:43:59.878 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:43:59.878 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:43:59.878 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888067 2024-08-19T15:43:59.878 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:43:59.878 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-38654705693 2024-08-19T15:43:59.878 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:43:59.879 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:43:59.879 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-38654705693 2024-08-19T15:43:59.879 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:43:59.879 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=38654705693 2024-08-19T15:43:59.879 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 38654705693' 2024-08-19T15:43:59.879 INFO:tasks.workunit.client.0.smithi102.stderr:466055: waiting osd.1 seq 38654705693 2024-08-19T15:43:59.879 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:43:59.879 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 38654705693 -lt 38654705693 2024-08-19T15:43:59.879 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:44:01.152 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //homee/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953475 2024-08-19T15:44:01.152 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:44:01.152 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:44:01.152 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953475 2024-08-19T15:44:01.152 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:44:01.152 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953475 2024-08-19T15:44:01.152 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953475' 2024-08-19T15:44:01.152 INFO:tasks.workunit.client.0.smithi102.stderr:466061: waiting osd.2 seq 137438953475 2024-08-19T15:44:01.152 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:44:01.152 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953476 -lt 137438953475 2024-08-19T15:44:01.153 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:44:01.153 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-141733920771 2024-08-19T15:44:01.153 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:44:01.153 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:44:01.153 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-141733920771 2024-08-19T15:44:01.153 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:44:01.153 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920771 2024-08-19T15:44:01.153 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 141733920771' 2024-08-19T15:44:01.153 INFO:tasks.workunit.client.0.smithi102.stderr:466061: waiting osd.3 seq 141733920771 2024-08-19T15:44:01.153 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:44:01.153 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920772 -lt 141733920771 2024-08-19T15:44:01.153 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:44:01.153 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:44:01.154 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:44:01.154 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:44:01.154 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:44:01.154 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:44:01.154 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:44:01.154 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:44:01.154 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:44:01.154 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:44:01.154 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:44:01.154 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:44:01.154 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:44:01.168 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/quntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953474 2024-08-19T15:44:01.168 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:44:01.168 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:44:01.168 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953474 2024-08-19T15:44:01.168 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:44:01.168 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-08-19T15:44:01.169 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953474' 2024-08-19T15:44:01.169 INFO:tasks.workunit.client.0.smithi102.stderr:466058: waiting osd.2 seq 137438953474 2024-08-19T15:44:01.169 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:44:01.169 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953476 -lt 137438953474 2024-08-19T15:44:01.169 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:44:01.169 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-141733920770 2024-08-19T15:44:01.169 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:44:01.169 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:44:01.169 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-141733920770 2024-08-19T15:44:01.169 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:44:01.169 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920770 2024-08-19T15:44:01.169 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 141733920770' 2024-08-19T15:44:01.169 INFO:tasks.workunit.client.0.smithi102.stderr:466058: waiting osd.3 seq 141733920770 2024-08-19T15:44:01.170 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:44:01.170 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920772 -lt 141733920770 2024-08-19T15:44:01.170 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:44:01.170 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:44:01.170 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:44:01.170 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:44:01.170 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:44:01.170 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:44:01.170 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:44:01.170 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:44:01.170 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:44:01.170 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:44:01.170 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:44:01.171 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:44:01.171 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:44:01.227 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/st/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953476 2024-08-19T15:44:01.227 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:44:01.227 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:44:01.227 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953476 2024-08-19T15:44:01.227 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:44:01.227 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953476 2024-08-19T15:44:01.227 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953476' 2024-08-19T15:44:01.227 INFO:tasks.workunit.client.0.smithi102.stderr:466055: waiting osd.2 seq 137438953476 2024-08-19T15:44:01.227 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:44:01.227 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953476 -lt 137438953476 2024-08-19T15:44:01.228 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:44:01.228 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-141733920772 2024-08-19T15:44:01.228 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:44:01.228 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-19T15:44:01.228 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-141733920772 2024-08-19T15:44:01.228 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:44:01.228 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920772 2024-08-19T15:44:01.228 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 141733920772' 2024-08-19T15:44:01.228 INFO:tasks.workunit.client.0.smithi102.stderr:466055: waiting osd.3 seq 141733920772 2024-08-19T15:44:01.228 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T15:44:01.228 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920772 -lt 141733920772 2024-08-19T15:44:01.228 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:44:01.228 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:44:01.229 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:44:01.229 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:44:01.229 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:44:01.229 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:44:01.229 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:44:01.229 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:44:01.229 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:44:01.229 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:44:01.229 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:44:01.229 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:44:01.229 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:44:01.515 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qaandalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:44:01.515 INFO:tasks.workunit.client.0.smithi102.stderr:466058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:44:01.515 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:44:01.515 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:44:01.515 INFO:tasks.workunit.client.0.smithi102.stderr:466058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:44:01.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T15:44:01.561 INFO:tasks.workunit.client.0.smithi102.stderr:a/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:44:01.561 INFO:tasks.workunit.client.0.smithi102.stderr:466061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:44:01.561 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:44:01.561 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:44:01.561 INFO:tasks.workunit.client.0.smithi102.stderr:466061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:44:01.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T15:44:01.658 INFO:tasks.workunit.client.0.smithi102.stderr:/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:44:01.659 INFO:tasks.workunit.client.0.smithi102.stderr:466055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:44:01.659 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:44:01.659 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:44:01.659 INFO:tasks.workunit.client.0.smithi102.stderr:466055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:44:01.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-08-19T15:44:01.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T15:44:01.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 466053 2024-08-19T15:44:01.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-08-19T15:44:01.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 466056 2024-08-19T15:44:01.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-08-19T15:44:01.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-08-19T15:44:01.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-08-19T15:44:01.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: return_code=0 2024-08-19T15:44:01.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:882: unfound_erasure_coded: '[' 0 -ne 0 ']' 2024-08-19T15:44:01.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: get_pg ecpool SOMETHING 2024-08-19T15:44:01.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-08-19T15:44:01.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-08-19T15:44:01.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-08-19T15:44:01.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T15:44:01.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: local pg=1.0 2024-08-19T15:44:01.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: unfound_erasure_coded: repair 1.0 2024-08-19T15:44:01.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-08-19T15:44:01.929 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-08-19T15:44:01.929 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:44:01.929 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:44:01.929 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:44:01.929 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:44:02.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-08-19T15:43:41.878362+0000 2024-08-19T15:44:02.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-08-19T15:44:02.442 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-08-19T15:44:02.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-08-19T15:43:41.878362+0000 2024-08-19T15:44:02.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-08-19T15:44:02.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:43:41.878362+0000 2024-08-19T15:44:02.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:44:02.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:44:02.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:44:02.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:44:02.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:44:02.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:44:02.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:44:02.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:44:02.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:43:41.878362+0000 '>' 2024-08-19T15:43:41.878362+0000 2024-08-19T15:44:02.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:44:03.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:44:03.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:44:03.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:44:03.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:44:03.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:44:03.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:44:03.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:44:03.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:43:41.878362+0000 '>' 2024-08-19T15:43:41.878362+0000 2024-08-19T15:44:03.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:44:04.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:44:04.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:44:04.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:44:04.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:44:04.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:44:04.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:44:04.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:44:05.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:43:41.878362+0000 '>' 2024-08-19T15:43:41.878362+0000 2024-08-19T15:44:05.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:44:06.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:44:06.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:44:06.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:44:06.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:44:06.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:44:06.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:44:06.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:44:06.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:43:41.878362+0000 '>' 2024-08-19T15:43:41.878362+0000 2024-08-19T15:44:06.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:44:07.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:44:07.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:44:07.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:44:07.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:44:07.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:44:07.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:44:07.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:44:07.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:44:03.072248+0000 '>' 2024-08-19T15:43:41.878362+0000 2024-08-19T15:44:07.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:44:07.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: seq 1 60 2024-08-19T15:44:07.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: for f in `seq 1 60` 2024-08-19T15:44:07.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: ceph -s 2024-08-19T15:44:07.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: grep '1/1 objects unfound' 2024-08-19T15:44:08.145 INFO:tasks.workunit.client.0.smithi102.stdout: 1/1 objects unfound (100.000%) 2024-08-19T15:44:08.145 INFO:tasks.workunit.client.0.smithi102.stdout: 1/1 objects unfound (100.000%) 2024-08-19T15:44:08.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: break 2024-08-19T15:44:08.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: ceph -s 2024-08-19T15:44:08.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: grep '4 up' 2024-08-19T15:44:08.507 INFO:tasks.workunit.client.0.smithi102.stdout: osd: 4 osds: 4 up (since 12s), 4 in (since 60s) 2024-08-19T15:44:08.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: ceph -s 2024-08-19T15:44:08.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: grep '4 in' 2024-08-19T15:44:08.837 INFO:tasks.workunit.client.0.smithi102.stdout: osd: 4 osds: 4 up (since 12s), 4 in (since 60s) 2024-08-19T15:44:08.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: ceph -s 2024-08-19T15:44:08.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: grep '1/1 objects unfound' 2024-08-19T15:44:09.170 INFO:tasks.workunit.client.0.smithi102.stdout: 1/1 objects unfound (100.000%) 2024-08-19T15:44:09.170 INFO:tasks.workunit.client.0.smithi102.stdout: 1/1 objects unfound (100.000%) 2024-08-19T15:44:09.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-08-19T15:44:09.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:44:09.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:44:09.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:44:09.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:44:09.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:44:09.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:44:09.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:44:09.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:44:09.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:44:09.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:44:09.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:44:09.282 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:44:09.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:44:09.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:44:09.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:44:09.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:44:09.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:44:09.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:44:09.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:44:09.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:44:09.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:44:09.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:44:09.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:44:09.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:44:09.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:44:09.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:44:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:44:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:44:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-08-19T15:44:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-repair 0 2024-08-19T15:44:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-08-19T15:44:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-08-19T15:44:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-08-19T15:44:09.308 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:44:09.308 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:44:09.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:44:09.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:44:09.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:44:09.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:44:09.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:44:09.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:44:09.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:44:09.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:44:09.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:44:09.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:44:09.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:44:09.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:44:09.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:44:09.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:44:09.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:44:09.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-08-19T15:44:09.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-08-19T15:44:09.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:44:09.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:44:09.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.57280 2024-08-19T15:44:09.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.57280 2024-08-19T15:44:09.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:44:09.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:44:09.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-08-19T15:44:09.320 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-19T15:44:09.321 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-19T15:44:09.347 INFO:tasks.workunit:Running workunit scrub/osd-scrub-snaps.sh... 2024-08-19T15:44:09.348 DEBUG:teuthology.orchestra.run.smithi102:workunit test scrub/osd-scrub-snaps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=78215f5a1cf5eaf321d506a66539942e12851f34 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh 2024-08-19T15:44:09.410 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-19T15:44:09.417 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-19T15:44:09.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-08-19T15:44:09.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-08-19T15:44:09.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-08-19T15:44:09.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-08-19T15:44:09.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-08-19T15:44:09.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-08-19T15:44:09.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-08-19T15:44:09.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-08-19T15:44:09.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-snaps 2024-08-19T15:44:09.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:27: run: local dir=td/osd-scrub-snaps 2024-08-19T15:44:09.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:28: run: shift 2024-08-19T15:44:09.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: export CEPH_MON=127.0.0.1:7121 2024-08-19T15:44:09.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: CEPH_MON=127.0.0.1:7121 2024-08-19T15:44:09.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:31: run: export CEPH_ARGS 2024-08-19T15:44:09.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: uuidgen 2024-08-19T15:44:09.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: CEPH_ARGS+='--fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none ' 2024-08-19T15:44:09.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:33: run: CEPH_ARGS+='--mon-host=127.0.0.1:7121 ' 2024-08-19T15:44:09.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:35: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-08-19T15:44:09.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: set 2024-08-19T15:44:09.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-08-19T15:44:09.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: local 'funcs=TEST_scrub_snaps 2024-08-19T15:44:09.420 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_scrub_snaps_primary 2024-08-19T15:44:09.421 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_scrub_snaps_replica' 2024-08-19T15:44:09.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-08-19T15:44:09.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-08-19T15:44:09.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-snaps 2024-08-19T15:44:09.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-snaps 2024-08-19T15:44:09.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-08-19T15:44:09.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:44:09.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-08-19T15:44:09.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:44:09.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:44:09.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:44:09.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:44:09.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:44:09.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:44:09.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:44:09.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:44:09.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:44:09.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:44:09.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:44:09.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:44:09.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:44:09.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:44:09.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:44:09.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:44:09.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:44:09.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:44:09.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-08-19T15:44:09.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:44:09.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:44:09.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:44:09.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.470418 2024-08-19T15:44:09.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:44:09.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:44:09.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-snaps 2024-08-19T15:44:09.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T15:44:09.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:44:09.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:44:09.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.470418 2024-08-19T15:44:09.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T15:44:09.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-19T15:44:09.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-19T15:44:09.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T15:44:09.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-08-19T15:44:09.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps td/osd-scrub-snaps 2024-08-19T15:44:09.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:171: TEST_scrub_snaps: local dir=td/osd-scrub-snaps 2024-08-19T15:44:09.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:172: TEST_scrub_snaps: local poolname=test 2024-08-19T15:44:09.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:173: TEST_scrub_snaps: local OBJS=16 2024-08-19T15:44:09.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:174: TEST_scrub_snaps: local OSDS=1 2024-08-19T15:44:09.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:176: TEST_scrub_snaps: TESTDATA=testdata.470418 2024-08-19T15:44:09.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:178: TEST_scrub_snaps: run_mon td/osd-scrub-snaps a --osd_pool_default_size=1 2024-08-19T15:44:09.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-snaps 2024-08-19T15:44:09.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T15:44:09.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T15:44:09.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T15:44:09.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-snaps/a 2024-08-19T15:44:09.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=1 2024-08-19T15:44:09.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T15:44:09.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:44:09.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:44:09.812 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:44:09.812 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:44:09.812 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:44:09.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:44:09.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 2024-08-19T15:44:09.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T15:44:09.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T15:44:09.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:44:09.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:44:10.000 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T15:44:10.000 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T15:44:10.000 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:44:10.000 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:44:10.000 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:44:10.000 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:44:10.001 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:44:10.001 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:44:10.001 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.470418/ceph-mon.a.asok 2024-08-19T15:44:10.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:44:10.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.470418/ceph-mon.a.asok config get fsid 2024-08-19T15:44:10.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T15:44:10.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:44:10.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:44:10.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T15:44:10.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T15:44:10.082 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:44:10.082 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:44:10.082 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:44:10.082 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:44:10.082 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:44:10.082 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:44:10.082 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.470418/ceph-mon.a.asok 2024-08-19T15:44:10.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:44:10.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.470418/ceph-mon.a.asok config get mon_host 2024-08-19T15:44:10.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:179: TEST_scrub_snaps: run_mgr td/osd-scrub-snaps x 2024-08-19T15:44:10.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-snaps 2024-08-19T15:44:10.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T15:44:10.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T15:44:10.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T15:44:10.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-snaps/x 2024-08-19T15:44:10.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T15:44:10.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T15:44:10.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:44:10.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:44:10.955 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:44:10.956 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:44:10.956 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:44:10.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:44:10.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:44:10.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:44:10.982 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: expr 1 - 1 2024-08-19T15:44:10.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: seq 0 0 2024-08-19T15:44:10.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:44:10.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:182: TEST_scrub_snaps: run_osd td/osd-scrub-snaps 0 2024-08-19T15:44:10.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-08-19T15:44:10.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:44:10.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T15:44:10.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:44:10.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-08-19T15:44:10.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-08-19T15:44:10.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:44:10.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:44:10.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:44:10.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-08-19T15:44:10.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-08-19T15:44:10.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:44:10.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:44:10.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-08-19T15:44:10.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:44:10.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:44:10.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:44:10.988 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:44:10.988 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:44:10.988 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:44:10.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:44:10.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:44:10.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:44:10.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:44:10.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:44:10.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-08-19T15:44:10.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-08-19T15:44:10.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:44:10.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:44:10.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:44:10.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:44:10.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:44:10.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:44:10.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-08-19T15:44:10.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:44:10.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0e399943-735d-4af8-9155-8793d51dd086 2024-08-19T15:44:10.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0e399943-735d-4af8-9155-8793d51dd086' 2024-08-19T15:44:10.990 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 0e399943-735d-4af8-9155-8793d51dd086 2024-08-19T15:44:10.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:44:11.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBLaMNmagwrABAAKVWtEnNqPBrbS3Mckpm8tA== 2024-08-19T15:44:11.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBLaMNmagwrABAAKVWtEnNqPBrbS3Mckpm8tA=="}' 2024-08-19T15:44:11.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0e399943-735d-4af8-9155-8793d51dd086 -i td/osd-scrub-snaps/0/new.json 2024-08-19T15:44:11.186 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:44:11.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-08-19T15:44:11.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBLaMNmagwrABAAKVWtEnNqPBrbS3Mckpm8tA== --osd-uuid 0e399943-735d-4af8-9155-8793d51dd086 2024-08-19T15:44:11.229 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:44:11.227+0000 7f0dd112c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:44:11.230 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:44:11.229+0000 7f0dd112c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:44:11.231 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:44:11.230+0000 7f0dd112c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:44:11.232 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:44:11.231+0000 7f0dd112c640 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-08-19T15:44:11.232 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:44:11.231+0000 7f0dd112c640 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-08-19T15:44:11.232 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:44:11.231+0000 7f0dd112c640 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-08-19T15:44:11.321 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:44:11.320+0000 7f0dd112c640 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-08-19T15:44:16.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-08-19T15:44:16.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:44:16.836 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T15:44:16.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T15:44:16.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:44:17.182 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:44:17.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T15:44:17.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:44:17.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:44:17.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:44:17.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:44:17.207 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:44:17.206+0000 7f9b26daf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:44:17.209 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:44:17.208+0000 7f9b26daf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:44:17.210 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:44:17.209+0000 7f9b26daf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:44:17.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T15:44:17.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:44:17.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:44:17.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:44:17.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:44:17.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:44:17.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:44:17.448 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:44:17.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:44:17.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:44:17.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:44:17.730 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:44:17.729+0000 7f9b26daf640 -1 Falling back to public interface 2024-08-19T15:44:18.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:44:18.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:44:18.710 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:44:18.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:44:18.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:44:18.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:44:18.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:44:19.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:44:19.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:44:19.975 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:44:19.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:44:19.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:44:19.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:44:20.070 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:44:20.068+0000 7f9b26daf640 -1 osd.0 0 log_to_monitors true 2024-08-19T15:44:20.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:44:21.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:44:21.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:44:21.263 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:44:21.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:44:21.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:44:21.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:44:21.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:44:22.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:44:22.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:44:22.697 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:44:22.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:44:22.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:44:22.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:44:22.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:44:23.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:44:23.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:44:23.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:44:23.835 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:44:23.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:44:23.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:44:24.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:44:25.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:44:25.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:44:25.108 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:44:25.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:44:25.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:44:25.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:44:25.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:44:26.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:44:26.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:44:26.380 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:44:26.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:44:26.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:44:26.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:44:26.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:44:27.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:44:27.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:44:27.658 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:44:27.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:44:27.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:44:27.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:44:27.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:44:28.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:44:28.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:44:28.999 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:44:28.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:44:29.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:44:29.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:44:29.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:44:30.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:44:30.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:44:30.276 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:44:30.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:44:30.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:44:30.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:44:30.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:44:31.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:44:31.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:44:31.550 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:44:31.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:44:31.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:44:31.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:44:32.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:44:33.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:44:33.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:44:33.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:44:33.031 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:44:33.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:44:33.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:44:33.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:44:34.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:44:34.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:44:34.343 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:44:34.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:44:34.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:44:34.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:44:34.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:44:35.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:44:35.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:44:35.823 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:44:35.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:44:35.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:44:35.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:44:36.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:44:37.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:44:37.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:44:37.464 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:44:37.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:44:37.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:44:37.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:44:38.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:44:39.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:44:39.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:44:39.175 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:44:39.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:44:39.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:44:39.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:44:39.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:44:40.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:44:40.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:44:40.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:44:40.492 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:44:40.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:44:40.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:44:41.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:44:42.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:44:42.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:44:42.089 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:44:42.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:44:42.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:44:42.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:44:42.424 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3033011457,v1:127.0.0.1:6803/3033011457] [v2:127.0.0.1:6804/3033011457,v1:127.0.0.1:6805/3033011457] exists,up 0e399943-735d-4af8-9155-8793d51dd086 2024-08-19T15:44:42.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:44:42.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:44:42.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:44:42.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:186: TEST_scrub_snaps: ceph osd set noscrub 2024-08-19T15:44:42.849 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:44:42.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:187: TEST_scrub_snaps: ceph osd set nodeep-scrub 2024-08-19T15:44:43.250 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:44:43.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:190: TEST_scrub_snaps: create_pool test 1 1 2024-08-19T15:44:43.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-08-19T15:44:43.652 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' created 2024-08-19T15:44:43.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:44:44.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:191: TEST_scrub_snaps: wait_for_clean 2024-08-19T15:44:44.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:44:44.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:44:44.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:44:44.666 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:44:44.666 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:44:44.667 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:44:44.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:44:44.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:44:44.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:44:44.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:44:44.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:44:44.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:44:44.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:44:44.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:44:44.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:44:45.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-08-19T15:44:45.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:44:45.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:44:45.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:44:45.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836486 2024-08-19T15:44:45.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836486 2024-08-19T15:44:45.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486' 2024-08-19T15:44:45.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:44:45.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836486 2024-08-19T15:44:45.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:44:45.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:44:45.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836486 2024-08-19T15:44:45.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:44:45.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836486 2024-08-19T15:44:45.140 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836486 2024-08-19T15:44:45.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-08-19T15:44:45.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:44:45.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836486 2024-08-19T15:44:45.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:44:46.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:44:46.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:44:46.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836486 2024-08-19T15:44:46.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:44:47.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:44:47.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:44:47.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836486 -lt 21474836486 2024-08-19T15:44:47.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:44:47.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:44:47.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:44:48.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:44:48.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:44:48.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:44:48.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:44:48.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:44:48.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:44:48.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:44:48.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:44:48.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:44:48.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:44:48.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:44:48.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:44:48.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:44:48.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:44:48.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:44:48.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: ceph osd dump 2024-08-19T15:44:48.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: grep '^pool.*['\'']test['\'']' 2024-08-19T15:44:48.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: awk '{ print $2 }' 2024-08-19T15:44:49.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: poolid=1 2024-08-19T15:44:49.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:194: TEST_scrub_snaps: dd if=/dev/urandom of=testdata.470418 bs=1032 count=1 2024-08-19T15:44:49.159 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records in 2024-08-19T15:44:49.159 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records out 2024-08-19T15:44:49.159 INFO:tasks.workunit.client.0.smithi102.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.8856e-05 s, 21.1 MB/s 2024-08-19T15:44:49.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: seq 1 16 2024-08-19T15:44:49.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-08-19T15:44:49.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj1 testdata.470418 2024-08-19T15:44:49.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-08-19T15:44:49.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj2 testdata.470418 2024-08-19T15:44:49.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-08-19T15:44:49.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj3 testdata.470418 2024-08-19T15:44:49.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-08-19T15:44:49.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj4 testdata.470418 2024-08-19T15:44:49.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-08-19T15:44:49.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj5 testdata.470418 2024-08-19T15:44:49.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-08-19T15:44:49.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj6 testdata.470418 2024-08-19T15:44:49.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-08-19T15:44:49.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj7 testdata.470418 2024-08-19T15:44:49.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-08-19T15:44:49.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj8 testdata.470418 2024-08-19T15:44:49.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-08-19T15:44:49.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj9 testdata.470418 2024-08-19T15:44:49.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-08-19T15:44:49.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj10 testdata.470418 2024-08-19T15:44:50.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-08-19T15:44:50.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj11 testdata.470418 2024-08-19T15:44:50.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-08-19T15:44:50.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj12 testdata.470418 2024-08-19T15:44:50.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-08-19T15:44:50.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj13 testdata.470418 2024-08-19T15:44:50.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-08-19T15:44:50.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj14 testdata.470418 2024-08-19T15:44:50.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-08-19T15:44:50.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj15 testdata.470418 2024-08-19T15:44:50.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-08-19T15:44:50.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj16 testdata.470418 2024-08-19T15:44:50.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: get_primary test obj1 2024-08-19T15:44:50.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-08-19T15:44:50.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-08-19T15:44:50.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-08-19T15:44:50.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:44:50.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: local primary=0 2024-08-19T15:44:50.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:202: TEST_scrub_snaps: create_scenario td/osd-scrub-snaps test testdata.470418 0 2024-08-19T15:44:50.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-08-19T15:44:50.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-08-19T15:44:50.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.470418 2024-08-19T15:44:50.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2024-08-19T15:44:50.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-08-19T15:44:50.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-08-19T15:44:50.861 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap1 2024-08-19T15:44:50.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=1 2024-08-19T15:44:50.864 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records in 2024-08-19T15:44:50.864 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records out 2024-08-19T15:44:50.864 INFO:tasks.workunit.client.0.smithi102.stderr:256 bytes copied, 6.2084e-05 s, 4.1 MB/s 2024-08-19T15:44:50.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.470418 2024-08-19T15:44:50.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.470418 2024-08-19T15:44:51.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.470418 2024-08-19T15:44:51.282 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-08-19T15:44:51.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:44:51.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.470418 2024-08-19T15:44:51.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:44:51.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.470418 2024-08-19T15:44:51.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:44:51.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.470418 2024-08-19T15:44:51.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:44:51.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.470418 2024-08-19T15:44:51.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:44:51.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.470418 2024-08-19T15:44:51.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:44:51.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.470418 2024-08-19T15:44:51.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:44:51.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.470418 2024-08-19T15:44:51.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:44:51.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.470418 2024-08-19T15:44:51.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:44:51.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.470418 2024-08-19T15:44:51.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-08-19T15:44:51.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-08-19T15:44:52.117 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap2 2024-08-19T15:44:52.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=2 2024-08-19T15:44:52.120 INFO:tasks.workunit.client.0.smithi102.stderr:2+0 records in 2024-08-19T15:44:52.120 INFO:tasks.workunit.client.0.smithi102.stderr:2+0 records out 2024-08-19T15:44:52.120 INFO:tasks.workunit.client.0.smithi102.stderr:512 bytes copied, 6.2802e-05 s, 8.2 MB/s 2024-08-19T15:44:52.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.470418 2024-08-19T15:44:52.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-08-19T15:44:52.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-08-19T15:44:52.443 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap3 2024-08-19T15:44:52.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=3 2024-08-19T15:44:52.446 INFO:tasks.workunit.client.0.smithi102.stderr:3+0 records in 2024-08-19T15:44:52.446 INFO:tasks.workunit.client.0.smithi102.stderr:3+0 records out 2024-08-19T15:44:52.446 INFO:tasks.workunit.client.0.smithi102.stderr:768 bytes copied, 6.4341e-05 s, 11.9 MB/s 2024-08-19T15:44:52.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.470418 2024-08-19T15:44:52.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-08-19T15:44:52.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-08-19T15:44:52.753 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap4 2024-08-19T15:44:52.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=4 2024-08-19T15:44:52.756 INFO:tasks.workunit.client.0.smithi102.stderr:4+0 records in 2024-08-19T15:44:52.756 INFO:tasks.workunit.client.0.smithi102.stderr:4+0 records out 2024-08-19T15:44:52.756 INFO:tasks.workunit.client.0.smithi102.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 7.1236e-05 s, 14.4 MB/s 2024-08-19T15:44:52.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.470418 2024-08-19T15:44:52.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.470418 2024-08-19T15:44:53.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-08-19T15:44:53.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-08-19T15:44:53.289 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap5 2024-08-19T15:44:53.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-08-19T15:44:53.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-08-19T15:44:53.532 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap6 2024-08-19T15:44:53.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=6 2024-08-19T15:44:53.535 INFO:tasks.workunit.client.0.smithi102.stderr:6+0 records in 2024-08-19T15:44:53.535 INFO:tasks.workunit.client.0.smithi102.stderr:6+0 records out 2024-08-19T15:44:53.535 INFO:tasks.workunit.client.0.smithi102.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 6.6962e-05 s, 22.9 MB/s 2024-08-19T15:44:53.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.470418 2024-08-19T15:44:53.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-08-19T15:44:53.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-08-19T15:44:53.867 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap7 2024-08-19T15:44:53.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-08-19T15:44:53.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-08-19T15:44:54.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-08-19T15:44:54.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-08-19T15:44:54.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:44:54.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:44:54.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:44:54.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:44:54.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:44:54.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:44:54.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2024-08-19T15:44:55.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:44:55.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-08-19T15:44:56.651 INFO:tasks.workunit.client.0.smithi102.stdout:WARNING: only removing head with clones present 2024-08-19T15:44:56.652 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:ff7b1f36:::obj1:head# 2024-08-19T15:44:57.166 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-08-19T15:44:57.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-08-19T15:44:58.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:44:58.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-08-19T15:44:58.596 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:c52c9666:::obj5:2# 2024-08-19T15:44:59.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-08-19T15:44:59.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-08-19T15:45:00.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:45:00.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:45:00.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-08-19T15:45:00.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-08-19T15:45:00.923 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:45:00.923 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-08-19T15:45:00.923 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:00.923 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-08-19T15:45:00.923 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-08-19T15:45:00.923 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-08-19T15:45:00.923 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-08-19T15:45:00.924 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-08-19T15:45:00.924 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-08-19T15:45:00.924 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-08-19T15:45:00.924 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:45:00.924 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-08-19T15:45:00.924 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-08-19T15:45:00.924 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-08-19T15:45:00.924 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-08-19T15:45:00.924 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:45:00.924 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:45:00.924 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:00.925 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:45:00.925 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:45:00.925 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:45:00.925 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:00.925 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:45:00.925 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:45:00.925 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:45:00.925 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:00.925 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:45:00.925 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:45:00.925 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:45:00.926 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:00.926 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:45:00.926 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:45:00.926 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:45:00.926 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:00.926 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:45:00.926 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:45:00.926 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:45:00.926 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:00.926 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:45:00.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-08-19T15:45:00.927 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-08-19T15:45:00.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-08-19T15:45:01.485 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:c52c9666:::obj5:1# 2024-08-19T15:45:02.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-08-19T15:45:03.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-08-19T15:45:03.043 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:45:03.043 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-08-19T15:45:03.043 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:03.043 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-08-19T15:45:03.043 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-08-19T15:45:03.043 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-08-19T15:45:03.043 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-08-19T15:45:03.043 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-08-19T15:45:03.044 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-08-19T15:45:03.044 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-08-19T15:45:03.044 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:45:03.044 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-08-19T15:45:03.044 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-08-19T15:45:03.044 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-08-19T15:45:03.044 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-08-19T15:45:03.044 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:45:03.044 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:45:03.044 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:03.045 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:45:03.045 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:45:03.045 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:45:03.045 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:03.045 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:45:03.045 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:45:03.045 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:45:03.045 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:03.045 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:45:03.045 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:45:03.045 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:45:03.045 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:03.046 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:45:03.046 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:45:03.046 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:45:03.046 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:03.046 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:45:03.046 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:45:03.046 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:45:03.046 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:03.046 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:45:03.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-08-19T15:45:03.047 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-08-19T15:45:03.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-08-19T15:45:03.047 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-08-19T15:45:03.047 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-08-19T15:45:03.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:45:03.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=18 2024-08-19T15:45:03.835 INFO:tasks.workunit.client.0.smithi102.stderr:18+0 records in 2024-08-19T15:45:03.835 INFO:tasks.workunit.client.0.smithi102.stderr:18+0 records out 2024-08-19T15:45:03.835 INFO:tasks.workunit.client.0.smithi102.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000132653 s, 34.7 MB/s 2024-08-19T15:45:03.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.470418 2024-08-19T15:45:04.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2024-08-19T15:45:06.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:45:06.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=15 2024-08-19T15:45:06.400 INFO:tasks.workunit.client.0.smithi102.stderr:15+0 records in 2024-08-19T15:45:06.400 INFO:tasks.workunit.client.0.smithi102.stderr:15+0 records out 2024-08-19T15:45:06.400 INFO:tasks.workunit.client.0.smithi102.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000100164 s, 38.3 MB/s 2024-08-19T15:45:06.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.470418 2024-08-19T15:45:07.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2024-08-19T15:45:07.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-08-19T15:45:08.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:45:08.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-08-19T15:45:08.924 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:0ee9ae15:::obj4:7# 2024-08-19T15:45:09.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-08-19T15:45:10.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-08-19T15:45:10.479 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-08-19T15:45:10.479 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:10.479 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-08-19T15:45:10.479 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-08-19T15:45:10.479 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-08-19T15:45:10.479 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-08-19T15:45:10.479 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-08-19T15:45:10.479 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-08-19T15:45:10.480 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-08-19T15:45:10.480 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:45:10.480 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-08-19T15:45:10.480 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-08-19T15:45:10.480 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-08-19T15:45:10.480 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-08-19T15:45:10.480 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:45:10.480 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:10.480 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:45:10.480 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:45:10.480 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:10.481 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:45:10.481 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:45:10.481 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:10.481 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:45:10.481 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:45:10.481 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:10.481 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:45:10.481 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:45:10.481 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:10.481 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:45:10.481 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:45:10.481 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:10.481 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:45:10.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-08-19T15:45:10.482 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:10.482 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:10.482 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:10.482 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:10.482 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:10.482 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:10.483 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:45:10.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2024-08-19T15:45:10.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-08-19T15:45:11.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:45:11.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-08-19T15:45:11.787 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:461f8b5e:::obj16:7# 2024-08-19T15:45:12.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-08-19T15:45:13.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-08-19T15:45:13.346 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-08-19T15:45:13.346 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-08-19T15:45:13.346 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-08-19T15:45:13.346 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-08-19T15:45:13.346 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-08-19T15:45:13.346 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-08-19T15:45:13.346 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-08-19T15:45:13.346 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-08-19T15:45:13.346 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:45:13.347 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-08-19T15:45:13.347 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-08-19T15:45:13.347 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-08-19T15:45:13.347 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-08-19T15:45:13.347 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:45:13.347 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:45:13.347 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:45:13.347 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:45:13.347 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:45:13.347 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:45:13.347 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:45:13.347 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:45:13.348 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:45:13.348 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:45:13.348 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:45:13.348 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:45:13.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-08-19T15:45:13.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-08-19T15:45:13.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2024-08-19T15:45:14.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:45:14.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-08-19T15:45:15.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:45:15.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-08-19T15:45:15.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:45:15.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=7 2024-08-19T15:45:15.191 INFO:tasks.workunit.client.0.smithi102.stderr:7+0 records in 2024-08-19T15:45:15.192 INFO:tasks.workunit.client.0.smithi102.stderr:7+0 records out 2024-08-19T15:45:15.192 INFO:tasks.workunit.client.0.smithi102.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 8.812e-05 s, 20.3 MB/s 2024-08-19T15:45:15.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.470418 2024-08-19T15:45:16.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2024-08-19T15:45:17.256 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:45:17.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:45:17.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-08-19T15:45:18.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2024-08-19T15:45:19.377 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:45:20.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:45:20.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-08-19T15:45:21.196 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2024-08-19T15:45:21.822 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:45:22.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:45:22.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-08-19T15:45:23.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2024-08-19T15:45:23.784 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:45:24.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:45:24.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-08-19T15:45:25.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2024-08-19T15:45:26.322 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:45:26.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:45:26.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-08-19T15:45:27.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2024-08-19T15:45:28.361 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:45:28.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:45:28.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-08-19T15:45:29.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2024-08-19T15:45:30.470 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:45:30.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:45:30.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-08-19T15:45:31.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2024-08-19T15:45:32.440 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:45:32.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:45:32.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-08-19T15:45:33.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2024-08-19T15:45:34.483 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:45:35.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:45:35.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-08-19T15:45:36.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-08-19T15:45:36.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2024-08-19T15:45:36.656 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:45:36.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:45:36.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-08-19T15:45:37.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-08-19T15:45:37.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-08-19T15:45:37.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:204: TEST_scrub_snaps: rm -f testdata.470418 2024-08-19T15:45:37.958 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: expr 1 - 1 2024-08-19T15:45:37.959 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: seq 0 0 2024-08-19T15:45:37.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:45:37.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:208: TEST_scrub_snaps: activate_osd td/osd-scrub-snaps 0 2024-08-19T15:45:37.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-08-19T15:45:37.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:45:37.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:45:37.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:45:37.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-08-19T15:45:37.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-08-19T15:45:37.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:45:37.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:45:37.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:45:37.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-08-19T15:45:37.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-08-19T15:45:37.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:45:37.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:45:37.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-08-19T15:45:37.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:45:37.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:45:37.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:45:37.965 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:45:37.965 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:45:37.966 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:45:37.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:45:37.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:45:37.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:45:37.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-08-19T15:45:37.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-08-19T15:45:37.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:45:37.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:45:37.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:45:37.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:45:37.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:45:37.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:45:37.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-08-19T15:45:37.967 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:45:37.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:45:37.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:45:37.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-08-19T15:45:37.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:45:37.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:45:37.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:45:37.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:45:37.991 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:45:37.990+0000 7fa6ebfb1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:45:37.994 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:45:37.993+0000 7fa6ebfb1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:45:37.995 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:45:37.994+0000 7fa6ebfb1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:45:38.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:45:38.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:45:38.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:45:38.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:45:38.241 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:45:38.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:45:38.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:45:38.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:45:38.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:45:38.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:45:38.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:45:38.511 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:45:38.510+0000 7fa6ebfb1640 -1 Falling back to public interface 2024-08-19T15:45:39.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:45:39.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:45:39.505 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:45:39.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:45:39.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:45:39.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:45:39.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:45:40.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:45:40.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:45:40.774 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:45:40.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:45:40.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:45:40.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:45:40.795 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:45:40.793+0000 7fa6ebfb1640 -1 osd.0 18 log_to_monitors true 2024-08-19T15:45:41.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:45:42.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:45:42.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:45:42.066 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:45:42.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:45:42.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:45:42.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:45:42.336 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 21 up_thru 21 down_at 19 last_clean_interval [5,18) [v2:127.0.0.1:6802/3331307510,v1:127.0.0.1:6803/3331307510] [v2:127.0.0.1:6804/3331307510,v1:127.0.0.1:6805/3331307510] exists,up 0e399943-735d-4af8-9155-8793d51dd086 2024-08-19T15:45:42.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:45:42.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:45:42.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:45:42.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:210: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 25 2024-08-19T15:45:42.431 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:45:42.431 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-08-19T15:45:42.431 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:45:42.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:211: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 5 2024-08-19T15:45:42.533 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:45:42.534 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-08-19T15:45:42.534 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:45:42.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:212: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2024-08-19T15:45:42.641 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:45:42.641 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-08-19T15:45:42.641 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:45:42.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:213: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2024-08-19T15:45:42.744 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:45:42.744 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-08-19T15:45:42.744 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:45:42.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:216: TEST_scrub_snaps: wait_for_clean 2024-08-19T15:45:42.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:45:42.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:45:42.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:45:42.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:45:42.754 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:45:42.754 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:45:42.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:45:42.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:45:42.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:45:42.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:45:42.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:45:42.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:45:42.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:45:42.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:45:42.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:45:43.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-08-19T15:45:43.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:45:43.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:45:43.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:45:43.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=90194313218 2024-08-19T15:45:43.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 90194313218 2024-08-19T15:45:43.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-90194313218' 2024-08-19T15:45:43.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:45:43.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-90194313218 2024-08-19T15:45:43.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:45:43.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:45:43.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-90194313218 2024-08-19T15:45:43.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:45:43.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=90194313218 2024-08-19T15:45:43.215 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 90194313218 2024-08-19T15:45:43.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 90194313218' 2024-08-19T15:45:43.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:45:43.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 90194313218 2024-08-19T15:45:43.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:45:44.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:45:44.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:45:44.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 90194313218 2024-08-19T15:45:44.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:45:45.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:45:45.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:45:46.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 90194313218 -lt 90194313218 2024-08-19T15:45:46.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:45:46.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:45:46.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:45:46.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:45:46.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:45:46.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:45:46.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:45:46.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:45:46.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:45:46.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:45:46.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:45:46.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:45:46.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:45:46.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:45:46.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:45:47.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:45:47.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:45:47.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:45:47.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:218: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_shallow_scrub_chunk_max 2024-08-19T15:45:47.112 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:45:47.112 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_shallow_scrub_chunk_max": "25" 2024-08-19T15:45:47.112 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:45:47.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:219: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_shallow_scrub_chunk_min 2024-08-19T15:45:47.213 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:45:47.213 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_shallow_scrub_chunk_min": "5" 2024-08-19T15:45:47.213 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:45:47.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:220: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_pg_stat_report_interval_max_seconds 2024-08-19T15:45:47.313 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:45:47.313 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_pg_stat_report_interval_max_seconds": "1" 2024-08-19T15:45:47.313 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:45:47.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:221: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_pg_stat_report_interval_max_epochs 2024-08-19T15:45:47.414 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:45:47.414 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_pg_stat_report_interval_max_epochs": "1" 2024-08-19T15:45:47.414 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:45:47.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:222: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_scrub_chunk_max 2024-08-19T15:45:47.516 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:45:47.516 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_chunk_max": "25" 2024-08-19T15:45:47.517 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:45:47.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:223: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_scrub_chunk_min 2024-08-19T15:45:47.617 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:45:47.617 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_chunk_min": "5" 2024-08-19T15:45:47.617 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:45:47.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:225: TEST_scrub_snaps: local pgid=1.0 2024-08-19T15:45:47.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:226: TEST_scrub_snaps: pg_scrub 1.0 2024-08-19T15:45:47.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=1.0 2024-08-19T15:45:47.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 1.0 2024-08-19T15:45:47.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:45:47.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:45:47.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:45:47.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:45:47.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-08-19T15:44:43.567949+0000 2024-08-19T15:45:47.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 1.0 2024-08-19T15:45:48.133 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 1.0 on osd.0 to scrub 2024-08-19T15:45:48.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 1.0 2024-08-19T15:44:43.567949+0000 2024-08-19T15:45:48.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-08-19T15:45:48.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:44:43.567949+0000 2024-08-19T15:45:48.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:45:48.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:45:48.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:45:48.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:45:48.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:45:48.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:45:48.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:45:48.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:45:48.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:44:43.567949+0000 '>' 2024-08-19T15:44:43.567949+0000 2024-08-19T15:45:48.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:45:49.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:45:49.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:45:49.403 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:45:49.403 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:45:49.403 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:45:49.403 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:45:49.403 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:45:49.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:44:43.567949+0000 '>' 2024-08-19T15:44:43.567949+0000 2024-08-19T15:45:49.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:45:50.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:45:50.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:45:50.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:45:50.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:45:50.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:45:50.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:45:50.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:45:50.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:44:43.567949+0000 '>' 2024-08-19T15:44:43.567949+0000 2024-08-19T15:45:50.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:45:51.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:45:51.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:45:51.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:45:51.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:45:51.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:45:51.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:45:51.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:45:52.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:44:43.567949+0000 '>' 2024-08-19T15:44:43.567949+0000 2024-08-19T15:45:52.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:45:53.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:45:53.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:45:53.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:45:53.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:45:53.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:45:53.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:45:53.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:45:53.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:45:48.924983+0000 '>' 2024-08-19T15:44:43.567949+0000 2024-08-19T15:45:53.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:45:53.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:45:53.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: wc -l 2024-08-19T15:45:53.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: test 2 = 2 2024-08-19T15:45:53.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:232: TEST_scrub_snaps: rados list-inconsistent-pg test 2024-08-19T15:45:53.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:234: TEST_scrub_snaps: jq '. | length' td/osd-scrub-snaps/json 2024-08-19T15:45:53.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:234: TEST_scrub_snaps: test 1 = 1 2024-08-19T15:45:53.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:236: TEST_scrub_snaps: jq -r '.[0]' td/osd-scrub-snaps/json 2024-08-19T15:45:53.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:236: TEST_scrub_snaps: test 1.0 = 1.0 2024-08-19T15:45:53.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:238: TEST_scrub_snaps: rados list-inconsistent-obj 1.0 2024-08-19T15:45:53.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:243: TEST_scrub_snaps: jq .inconsistents 2024-08-19T15:45:53.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:243: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-08-19T15:45:53.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:250: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2024-08-19T15:45:53.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:250: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-08-19T15:45:53.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:251: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-08-19T15:45:53.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-08-19T15:45:53.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:253: TEST_scrub_snaps: rados list-inconsistent-snapset 1.0 2024-08-19T15:45:53.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:255: TEST_scrub_snaps: jq .inconsistents 2024-08-19T15:45:53.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:255: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-08-19T15:45:53.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2024-08-19T15:45:53.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-08-19T15:45:53.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:635: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-08-19T15:45:53.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-08-19T15:45:53.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:636: TEST_scrub_snaps: test no = yes 2024-08-19T15:45:53.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:641: TEST_scrub_snaps: test '' = yes 2024-08-19T15:45:53.656 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: find td/osd-scrub-snaps 2024-08-19T15:45:53.656 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: grep 'osd[^/]*\.pid' 2024-08-19T15:45:53.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: pidfiles=td/osd-scrub-snaps/osd.0.pid 2024-08-19T15:45:53.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:647: TEST_scrub_snaps: pids= 2024-08-19T15:45:53.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:648: TEST_scrub_snaps: for pidfile in ${pidfiles} 2024-08-19T15:45:53.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:650: TEST_scrub_snaps: cat td/osd-scrub-snaps/osd.0.pid 2024-08-19T15:45:53.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:650: TEST_scrub_snaps: pids+='489314 ' 2024-08-19T15:45:53.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: ERRORS=0 2024-08-19T15:45:53.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: seq 1 7 2024-08-19T15:45:53.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-08-19T15:45:53.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap1 2024-08-19T15:45:53.788 INFO:tasks.workunit.client.0.smithi102.stdout:removed pool test snap snap1 2024-08-19T15:45:53.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-08-19T15:45:53.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap2 2024-08-19T15:45:54.088 INFO:tasks.workunit.client.0.smithi102.stdout:removed pool test snap snap2 2024-08-19T15:45:54.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-08-19T15:45:54.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap3 2024-08-19T15:45:54.273 INFO:tasks.workunit.client.0.smithi102.stdout:removed pool test snap snap3 2024-08-19T15:45:54.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-08-19T15:45:54.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap4 2024-08-19T15:45:54.466 INFO:tasks.workunit.client.0.smithi102.stdout:removed pool test snap snap4 2024-08-19T15:45:54.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-08-19T15:45:54.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap5 2024-08-19T15:45:54.675 INFO:tasks.workunit.client.0.smithi102.stdout:removed pool test snap snap5 2024-08-19T15:45:54.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-08-19T15:45:54.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap6 2024-08-19T15:45:54.893 INFO:tasks.workunit.client.0.smithi102.stdout:removed pool test snap snap6 2024-08-19T15:45:54.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-08-19T15:45:54.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap7 2024-08-19T15:45:55.114 INFO:tasks.workunit.client.0.smithi102.stdout:removed pool test snap snap7 2024-08-19T15:45:55.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:659: TEST_scrub_snaps: sleep 5 2024-08-19T15:46:00.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:660: TEST_scrub_snaps: local -i loop=0 2024-08-19T15:46:00.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:661: TEST_scrub_snaps: ceph pg dump pgs 2024-08-19T15:46:00.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:661: TEST_scrub_snaps: grep -q snaptrim 2024-08-19T15:46:00.371 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:46:00.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:663: TEST_scrub_snaps: ceph pg dump pgs 2024-08-19T15:46:00.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:663: TEST_scrub_snaps: grep -q snaptrim_error 2024-08-19T15:46:00.633 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:46:00.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:665: TEST_scrub_snaps: break 2024-08-19T15:46:00.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:674: TEST_scrub_snaps: ceph pg dump pgs 2024-08-19T15:46:00.896 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T15:46:00.896 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 35 0 0 0 0 23416 0 0 60 0 60 active+clean+inconsistent+snaptrim_error 2024-08-19T15:45:55.135961+0000 23'60 29:138 [0] 0 [0] 0 18'56 2024-08-19T15:45:48.924983+0000 0'0 2024-08-19T15:44:43.567949+0000 7 1 periodic scrub scheduled @ 2024-08-21T01:29:57.603934+0000 33 0 2024-08-19T15:46:00.896 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T15:46:00.896 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T15:46:00.896 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:46:00.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:676: TEST_scrub_snaps: for pid in $pids 2024-08-19T15:46:00.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:678: TEST_scrub_snaps: kill -0 489314 2024-08-19T15:46:00.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:685: TEST_scrub_snaps: kill_daemons td/osd-scrub-snaps 2024-08-19T15:46:00.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:46:00.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:46:00.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:46:00.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:46:00.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:46:06.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:46:06.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:687: TEST_scrub_snaps: declare -a err_strings 2024-08-19T15:46:06.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:688: TEST_scrub_snaps: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' 2024-08-19T15:46:06.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:689: TEST_scrub_snaps: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' 2024-08-19T15:46:06.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:690: TEST_scrub_snaps: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' 2024-08-19T15:46:06.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:691: TEST_scrub_snaps: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' 2024-08-19T15:46:06.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:692: TEST_scrub_snaps: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' 2024-08-19T15:46:06.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:693: TEST_scrub_snaps: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' 2024-08-19T15:46:06.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:694: TEST_scrub_snaps: err_strings[6]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' 2024-08-19T15:46:06.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:695: TEST_scrub_snaps: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' 2024-08-19T15:46:06.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:696: TEST_scrub_snaps: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' 2024-08-19T15:46:06.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:697: TEST_scrub_snaps: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' 2024-08-19T15:46:06.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:698: TEST_scrub_snaps: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' 2024-08-19T15:46:06.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:699: TEST_scrub_snaps: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' 2024-08-19T15:46:06.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' 2024-08-19T15:46:06.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:701: TEST_scrub_snaps: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' 2024-08-19T15:46:06.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' 2024-08-19T15:46:06.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:703: TEST_scrub_snaps: err_strings[15]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' 2024-08-19T15:46:06.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:704: TEST_scrub_snaps: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' 2024-08-19T15:46:06.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:705: TEST_scrub_snaps: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' 2024-08-19T15:46:06.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:706: TEST_scrub_snaps: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' 2024-08-19T15:46:06.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:707: TEST_scrub_snaps: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' 2024-08-19T15:46:06.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:708: TEST_scrub_snaps: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2024-08-19T15:46:06.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:709: TEST_scrub_snaps: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' 2024-08-19T15:46:06.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-08-19T15:46:06.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-08-19T15:46:06.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:46:06.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:721: TEST_scrub_snaps: '[' 0 '!=' 0 ']' 2024-08-19T15:46:06.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:727: TEST_scrub_snaps: echo 'TEST PASSED' 2024-08-19T15:46:06.271 INFO:tasks.workunit.client.0.smithi102.stdout:TEST PASSED 2024-08-19T15:46:06.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:728: TEST_scrub_snaps: return 0 2024-08-19T15:46:06.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-08-19T15:46:06.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-08-19T15:46:06.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:46:06.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-08-19T15:46:06.272 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:46:06.272 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:46:06.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:46:06.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:46:06.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:46:06.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:46:06.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:46:06.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:46:06.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:46:06.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:46:06.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:46:06.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:46:06.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:46:06.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:46:06.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:46:06.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:46:06.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:46:06.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:46:06.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-08-19T15:46:06.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:46:06.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:46:06.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:46:06.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.470418 2024-08-19T15:46:06.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:46:06.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:46:06.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-08-19T15:46:06.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-08-19T15:46:06.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-snaps 2024-08-19T15:46:06.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-snaps 2024-08-19T15:46:06.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-08-19T15:46:06.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:46:06.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-08-19T15:46:06.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:46:06.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:46:06.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:46:06.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:46:06.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:46:06.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:46:06.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:46:06.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:46:06.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:46:06.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:46:06.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:46:06.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:46:06.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:46:06.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:46:06.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:46:06.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:46:06.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:46:06.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:46:06.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-08-19T15:46:06.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:46:06.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:46:06.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:46:06.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.470418 2024-08-19T15:46:06.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:46:06.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:46:06.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-snaps 2024-08-19T15:46:06.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T15:46:06.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:46:06.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:46:06.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.470418 2024-08-19T15:46:06.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T15:46:06.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T15:46:06.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T15:46:06.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-08-19T15:46:06.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_primary td/osd-scrub-snaps 2024-08-19T15:46:06.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1177: TEST_scrub_snaps_primary: local dir=td/osd-scrub-snaps 2024-08-19T15:46:06.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1178: TEST_scrub_snaps_primary: ORIG_ARGS='--fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-08-19T15:46:06.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1179: TEST_scrub_snaps_primary: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-08-19T15:46:06.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1180: TEST_scrub_snaps_primary: _scrub_snaps_multi td/osd-scrub-snaps primary 2024-08-19T15:46:06.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2024-08-19T15:46:06.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:733: _scrub_snaps_multi: local poolname=test 2024-08-19T15:46:06.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:734: _scrub_snaps_multi: local OBJS=16 2024-08-19T15:46:06.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:735: _scrub_snaps_multi: local OSDS=2 2024-08-19T15:46:06.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: local which=primary 2024-08-19T15:46:06.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:738: _scrub_snaps_multi: TESTDATA=testdata.470418 2024-08-19T15:46:06.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2024-08-19T15:46:06.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-snaps 2024-08-19T15:46:06.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T15:46:06.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T15:46:06.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T15:46:06.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-snaps/a 2024-08-19T15:46:06.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2024-08-19T15:46:06.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T15:46:06.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:46:06.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:46:06.697 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:46:06.697 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:46:06.697 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:46:06.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:46:06.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-08-19T15:46:06.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T15:46:06.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T15:46:06.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:46:06.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:46:06.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T15:46:06.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T15:46:06.892 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:46:06.892 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:46:06.892 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:46:06.892 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:46:06.892 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:46:06.892 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:46:06.892 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.470418/ceph-mon.a.asok 2024-08-19T15:46:06.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:46:06.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.470418/ceph-mon.a.asok config get fsid 2024-08-19T15:46:06.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T15:46:06.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:46:06.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:46:06.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T15:46:06.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T15:46:06.972 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:46:06.972 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:46:06.972 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:46:06.972 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:46:06.972 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:46:06.972 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:46:06.972 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.470418/ceph-mon.a.asok 2024-08-19T15:46:06.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:46:06.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.470418/ceph-mon.a.asok config get mon_host 2024-08-19T15:46:07.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:741: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2024-08-19T15:46:07.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-snaps 2024-08-19T15:46:07.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T15:46:07.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T15:46:07.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T15:46:07.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-snaps/x 2024-08-19T15:46:07.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T15:46:07.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T15:46:07.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:46:07.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:46:07.866 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:46:07.866 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:46:07.866 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:46:07.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:46:07.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:46:07.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:46:07.902 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: expr 2 - 1 2024-08-19T15:46:07.903 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: seq 0 1 2024-08-19T15:46:07.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:46:07.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2024-08-19T15:46:07.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-08-19T15:46:07.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:46:07.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T15:46:07.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:46:07.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-08-19T15:46:07.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-08-19T15:46:07.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:46:07.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:46:07.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:46:07.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-08-19T15:46:07.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-08-19T15:46:07.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:46:07.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:46:07.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-08-19T15:46:07.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:46:07.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:46:07.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:46:07.909 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:46:07.909 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:46:07.909 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:46:07.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:46:07.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:46:07.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:46:07.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:46:07.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:46:07.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-08-19T15:46:07.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-08-19T15:46:07.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:46:07.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:46:07.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:46:07.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:46:07.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:46:07.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:46:07.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-08-19T15:46:07.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:46:07.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=35afcfe8-0dcc-411c-bb7d-4b4e3b58909d 2024-08-19T15:46:07.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 35afcfe8-0dcc-411c-bb7d-4b4e3b58909d' 2024-08-19T15:46:07.911 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 35afcfe8-0dcc-411c-bb7d-4b4e3b58909d 2024-08-19T15:46:07.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:46:07.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC/aMNma3uRNxAA5LiuslkrvEo9/HyVH21lMA== 2024-08-19T15:46:07.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC/aMNma3uRNxAA5LiuslkrvEo9/HyVH21lMA=="}' 2024-08-19T15:46:07.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 35afcfe8-0dcc-411c-bb7d-4b4e3b58909d -i td/osd-scrub-snaps/0/new.json 2024-08-19T15:46:08.172 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:46:08.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-08-19T15:46:08.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/aMNma3uRNxAA5LiuslkrvEo9/HyVH21lMA== --osd-uuid 35afcfe8-0dcc-411c-bb7d-4b4e3b58909d 2024-08-19T15:46:08.222 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:08.221+0000 7f6226e1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:46:08.224 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:08.223+0000 7f6226e1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:46:08.225 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:08.224+0000 7f6226e1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:46:08.226 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:08.224+0000 7f6226e1e640 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-08-19T15:46:08.226 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:08.224+0000 7f6226e1e640 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-08-19T15:46:08.226 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:08.224+0000 7f6226e1e640 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-08-19T15:46:08.315 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:08.313+0000 7f6226e1e640 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-08-19T15:46:13.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-08-19T15:46:13.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:46:13.572 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T15:46:13.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T15:46:13.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:46:13.930 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:46:13.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T15:46:13.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:46:13.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:46:13.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:46:13.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:46:13.968 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:13.967+0000 7f9138c24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:46:13.970 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:13.969+0000 7f9138c24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:46:13.971 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:13.970+0000 7f9138c24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:46:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T15:46:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:46:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:46:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:46:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:46:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:14.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:46:14.210 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:46:14.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:14.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:46:14.498 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:14.496+0000 7f9138c24640 -1 Falling back to public interface 2024-08-19T15:46:14.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:15.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:15.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:15.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:46:15.521 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:46:15.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:15.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:46:15.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:16.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:16.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:16.806 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:46:16.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:46:16.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:16.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:46:16.812 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:16.810+0000 7f9138c24640 -1 osd.0 0 log_to_monitors true 2024-08-19T15:46:17.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:18.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:18.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:18.095 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:46:18.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:46:18.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:18.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:46:18.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:19.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:19.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:19.382 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:46:19.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:46:19.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:19.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:46:19.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:20.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:20.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:20.713 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:46:20.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:46:20.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:20.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:46:20.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:21.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:22.001 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:46:22.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:22.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:46:22.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:22.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:46:22.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:23.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:23.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:23.286 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:46:23.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:46:23.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:23.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:46:23.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:24.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:24.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:24.574 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:46:24.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:46:24.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:24.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:46:24.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:25.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:25.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:25.863 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:46:25.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:46:25.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:25.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:46:26.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:27.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:27.155 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:46:27.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:27.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:46:27.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:27.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:46:27.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:28.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:28.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:28.471 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:46:28.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:46:28.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:28.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:46:29.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:30.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:30.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:30.024 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:46:30.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:46:30.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:30.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:46:30.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:31.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:31.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:31.753 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:46:31.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:46:31.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:31.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:46:32.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:33.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:33.498 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:46:33.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:33.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:46:33.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:33.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:46:33.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:34.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:34.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:34.998 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:46:34.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:46:34.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:34.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:46:35.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:36.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:36.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:36.404 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:46:36.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:46:36.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:36.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:46:37.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:38.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:38.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:38.191 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:46:38.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:46:38.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:38.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:46:38.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:39.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:39.558 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:46:39.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:39.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:46:39.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:39.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:46:39.910 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3038247879,v1:127.0.0.1:6803/3038247879] [v2:127.0.0.1:6804/3038247879,v1:127.0.0.1:6805/3038247879] exists,up 35afcfe8-0dcc-411c-bb7d-4b4e3b58909d 2024-08-19T15:46:39.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:46:39.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:46:39.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:46:39.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:46:39.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2024-08-19T15:46:39.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-08-19T15:46:39.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:46:39.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T15:46:39.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:46:39.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/1 2024-08-19T15:46:39.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-08-19T15:46:39.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:46:39.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:46:39.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:46:39.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-08-19T15:46:39.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-08-19T15:46:39.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:46:39.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:46:39.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-08-19T15:46:39.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:46:39.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:46:39.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:46:39.916 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:46:39.916 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:46:39.916 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:46:39.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:46:39.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:46:39.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:46:39.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:46:39.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:46:39.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-08-19T15:46:39.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-08-19T15:46:39.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:46:39.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:46:39.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:46:39.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:46:39.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:46:39.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:46:39.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/1 2024-08-19T15:46:39.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:46:39.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6d06ba29-0bfc-49cb-ad80-79014868d11b 2024-08-19T15:46:39.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6d06ba29-0bfc-49cb-ad80-79014868d11b' 2024-08-19T15:46:39.917 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 6d06ba29-0bfc-49cb-ad80-79014868d11b 2024-08-19T15:46:39.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:46:39.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDfaMNmuXH3NxAA8y+1IOS9i98VThdHGCP3TA== 2024-08-19T15:46:39.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDfaMNmuXH3NxAA8y+1IOS9i98VThdHGCP3TA=="}' 2024-08-19T15:46:39.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6d06ba29-0bfc-49cb-ad80-79014868d11b -i td/osd-scrub-snaps/1/new.json 2024-08-19T15:46:40.283 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:46:40.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/1/new.json 2024-08-19T15:46:40.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDfaMNmuXH3NxAA8y+1IOS9i98VThdHGCP3TA== --osd-uuid 6d06ba29-0bfc-49cb-ad80-79014868d11b 2024-08-19T15:46:40.335 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:40.334+0000 7fe45e243640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:46:40.376 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:40.374+0000 7fe45e243640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:46:40.377 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:40.375+0000 7fe45e243640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:46:40.377 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:40.376+0000 7fe45e243640 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-08-19T15:46:40.377 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:40.376+0000 7fe45e243640 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-08-19T15:46:40.377 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:40.376+0000 7fe45e243640 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-08-19T15:46:40.442 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:40.441+0000 7fe45e243640 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2024-08-19T15:46:46.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2024-08-19T15:46:46.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:46:46.145 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T15:46:46.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T15:46:46.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:46:46.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T15:46:46.498 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:46:46.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:46:46.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:46:46.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:46:46.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:46:46.536 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:46.535+0000 7f4c5d1fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:46:46.561 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:46.560+0000 7f4c5d1fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:46:46.562 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:46.561+0000 7f4c5d1fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:46:46.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T15:46:46.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:46:46.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:46:46.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:46:46.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:46:46.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:46.827 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:46:46.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:46:46.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:46.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:46:47.093 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:47.092+0000 7f4c5d1fc640 -1 Falling back to public interface 2024-08-19T15:46:47.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:48.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:48.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:48.148 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:46:48.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:46:48.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:48.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:46:48.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:49.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:49.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:49.434 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:46:49.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:46:49.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:49.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:46:49.439 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:46:49.438+0000 7f4c5d1fc640 -1 osd.1 0 log_to_monitors true 2024-08-19T15:46:49.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:50.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:50.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:50.894 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:46:50.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:46:50.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:50.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:46:51.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:52.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:52.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:52.184 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:46:52.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:46:52.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:52.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:46:52.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:53.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:53.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:53.477 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:46:53.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:46:53.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:53.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:46:53.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:54.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:54.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:54.763 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:46:54.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:46:54.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:54.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:46:55.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:56.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:56.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:56.054 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:46:56.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:46:56.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:56.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:46:56.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:57.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:57.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:57.388 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:46:57.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:46:57.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:57.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:46:57.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:58.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:58.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:58.677 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:46:58.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:46:58.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:58.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:46:58.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:46:59.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:46:59.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:46:59.990 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:46:59.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:46:59.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:46:59.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:47:00.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:47:01.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:47:01.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:47:01.279 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:47:01.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:47:01.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:47:01.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:47:01.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:47:02.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:47:02.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:47:02.571 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:47:02.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:47:02.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:47:02.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:47:03.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:47:04.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:47:04.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:47:04.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:47:04.088 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:47:04.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:47:04.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:47:04.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:47:05.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:47:05.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:47:05.464 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:47:05.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:47:05.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:47:05.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:47:05.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:47:06.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:47:06.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:47:06.976 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:47:06.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:47:06.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:47:06.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:47:07.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:47:08.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:47:08.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:47:08.577 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:47:08.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:47:08.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:47:08.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:47:09.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:47:10.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:47:10.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:47:10.313 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:47:10.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:47:10.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:47:10.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:47:11.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:47:12.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:47:12.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:47:12.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:47:12.125 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:47:12.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:47:12.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:47:12.422 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3484791459,v1:127.0.0.1:6811/3484791459] [v2:127.0.0.1:6812/3484791459,v1:127.0.0.1:6813/3484791459] exists,up 6d06ba29-0bfc-49cb-ad80-79014868d11b 2024-08-19T15:47:12.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:47:12.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:47:12.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:47:12.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:748: _scrub_snaps_multi: ceph osd set noscrub 2024-08-19T15:47:12.818 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:47:12.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:749: _scrub_snaps_multi: ceph osd set nodeep-scrub 2024-08-19T15:47:13.336 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:47:13.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: create_pool test 1 1 2024-08-19T15:47:13.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-08-19T15:47:13.746 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' created 2024-08-19T15:47:13.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:47:14.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:753: _scrub_snaps_multi: wait_for_clean 2024-08-19T15:47:14.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:47:14.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:47:14.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:47:14.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:47:14.760 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:47:14.760 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:47:14.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:47:14.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:47:14.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:47:14.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:47:14.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:47:14.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:47:14.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:47:14.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:47:14.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:47:15.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:47:15.134 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:47:15.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:47:15.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:47:15.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:47:15.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836494 2024-08-19T15:47:15.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836494 2024-08-19T15:47:15.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494' 2024-08-19T15:47:15.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:47:15.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:47:15.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672965 2024-08-19T15:47:15.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672965 2024-08-19T15:47:15.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494 1-42949672965' 2024-08-19T15:47:15.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:47:15.363 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836494 2024-08-19T15:47:15.363 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:47:15.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:47:15.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836494 2024-08-19T15:47:15.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:47:15.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836494 2024-08-19T15:47:15.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-08-19T15:47:15.365 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836494 2024-08-19T15:47:15.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:47:15.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836494 2024-08-19T15:47:15.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:47:16.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:47:16.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:47:16.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836494 2024-08-19T15:47:16.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:47:16.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672965 2024-08-19T15:47:16.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:47:16.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:47:16.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672965 2024-08-19T15:47:16.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:47:16.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672965 2024-08-19T15:47:16.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-08-19T15:47:16.928 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672965 2024-08-19T15:47:16.928 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:47:17.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672965 2024-08-19T15:47:17.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:47:17.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:47:17.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:47:17.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:47:17.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:47:17.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:47:17.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:47:17.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:47:17.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:47:17.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:47:17.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:47:17.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:47:17.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:47:17.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:47:17.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:47:18.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:47:18.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:47:18.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:47:18.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: ceph osd dump 2024-08-19T15:47:18.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2024-08-19T15:47:18.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: awk '{ print $2 }' 2024-08-19T15:47:18.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: poolid=1 2024-08-19T15:47:18.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:756: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.470418 bs=1032 count=1 2024-08-19T15:47:18.629 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records in 2024-08-19T15:47:18.629 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records out 2024-08-19T15:47:18.629 INFO:tasks.workunit.client.0.smithi102.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.5701e-05 s, 22.6 MB/s 2024-08-19T15:47:18.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: seq 1 16 2024-08-19T15:47:18.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:47:18.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj1 testdata.470418 2024-08-19T15:47:18.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:47:18.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj2 testdata.470418 2024-08-19T15:47:18.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:47:18.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj3 testdata.470418 2024-08-19T15:47:19.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:47:19.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj4 testdata.470418 2024-08-19T15:47:19.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:47:19.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj5 testdata.470418 2024-08-19T15:47:19.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:47:19.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj6 testdata.470418 2024-08-19T15:47:19.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:47:19.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj7 testdata.470418 2024-08-19T15:47:19.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:47:19.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj8 testdata.470418 2024-08-19T15:47:19.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:47:19.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj9 testdata.470418 2024-08-19T15:47:19.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:47:19.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj10 testdata.470418 2024-08-19T15:47:20.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:47:20.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj11 testdata.470418 2024-08-19T15:47:20.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:47:20.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj12 testdata.470418 2024-08-19T15:47:20.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:47:20.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj13 testdata.470418 2024-08-19T15:47:20.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:47:20.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj14 testdata.470418 2024-08-19T15:47:20.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:47:20.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj15 testdata.470418 2024-08-19T15:47:20.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:47:20.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj16 testdata.470418 2024-08-19T15:47:20.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: get_primary test obj1 2024-08-19T15:47:20.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-08-19T15:47:20.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-08-19T15:47:20.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-08-19T15:47:20.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:47:21.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: local primary=1 2024-08-19T15:47:21.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: get_not_primary test obj1 2024-08-19T15:47:21.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-08-19T15:47:21.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-08-19T15:47:21.135 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-08-19T15:47:21.135 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-08-19T15:47:21.136 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-08-19T15:47:21.136 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-08-19T15:47:21.136 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:47:21.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T15:47:21.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-08-19T15:47:21.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T15:47:21.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: local replica=0 2024-08-19T15:47:21.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.470418 '$primary' 2024-08-19T15:47:21.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.470418 1 2024-08-19T15:47:21.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-08-19T15:47:21.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-08-19T15:47:21.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.470418 2024-08-19T15:47:21.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=1 2024-08-19T15:47:21.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-08-19T15:47:21.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-08-19T15:47:21.948 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap1 2024-08-19T15:47:21.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=1 2024-08-19T15:47:21.950 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records in 2024-08-19T15:47:21.950 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records out 2024-08-19T15:47:21.951 INFO:tasks.workunit.client.0.smithi102.stderr:256 bytes copied, 5.9406e-05 s, 4.3 MB/s 2024-08-19T15:47:21.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.470418 2024-08-19T15:47:22.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.470418 2024-08-19T15:47:22.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.470418 2024-08-19T15:47:22.519 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-08-19T15:47:22.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:47:22.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.470418 2024-08-19T15:47:22.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:47:22.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.470418 2024-08-19T15:47:22.820 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:47:22.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.470418 2024-08-19T15:47:22.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:47:22.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.470418 2024-08-19T15:47:23.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:47:23.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.470418 2024-08-19T15:47:23.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:47:23.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.470418 2024-08-19T15:47:23.356 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:47:23.356 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.470418 2024-08-19T15:47:23.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:47:23.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.470418 2024-08-19T15:47:23.624 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:47:23.624 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.470418 2024-08-19T15:47:23.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-08-19T15:47:23.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-08-19T15:47:23.950 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap2 2024-08-19T15:47:23.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=2 2024-08-19T15:47:23.953 INFO:tasks.workunit.client.0.smithi102.stderr:2+0 records in 2024-08-19T15:47:23.953 INFO:tasks.workunit.client.0.smithi102.stderr:2+0 records out 2024-08-19T15:47:23.953 INFO:tasks.workunit.client.0.smithi102.stderr:512 bytes copied, 6.2208e-05 s, 8.2 MB/s 2024-08-19T15:47:23.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.470418 2024-08-19T15:47:24.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-08-19T15:47:24.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-08-19T15:47:24.377 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap3 2024-08-19T15:47:24.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=3 2024-08-19T15:47:24.380 INFO:tasks.workunit.client.0.smithi102.stderr:3+0 records in 2024-08-19T15:47:24.380 INFO:tasks.workunit.client.0.smithi102.stderr:3+0 records out 2024-08-19T15:47:24.380 INFO:tasks.workunit.client.0.smithi102.stderr:768 bytes copied, 6.756e-05 s, 11.4 MB/s 2024-08-19T15:47:24.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.470418 2024-08-19T15:47:24.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-08-19T15:47:24.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-08-19T15:47:24.787 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap4 2024-08-19T15:47:24.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=4 2024-08-19T15:47:24.790 INFO:tasks.workunit.client.0.smithi102.stderr:4+0 records in 2024-08-19T15:47:24.790 INFO:tasks.workunit.client.0.smithi102.stderr:4+0 records out 2024-08-19T15:47:24.790 INFO:tasks.workunit.client.0.smithi102.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 6.9902e-05 s, 14.6 MB/s 2024-08-19T15:47:24.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.470418 2024-08-19T15:47:25.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.470418 2024-08-19T15:47:25.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-08-19T15:47:25.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-08-19T15:47:25.290 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap5 2024-08-19T15:47:25.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-08-19T15:47:25.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-08-19T15:47:25.473 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap6 2024-08-19T15:47:25.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=6 2024-08-19T15:47:25.475 INFO:tasks.workunit.client.0.smithi102.stderr:6+0 records in 2024-08-19T15:47:25.476 INFO:tasks.workunit.client.0.smithi102.stderr:6+0 records out 2024-08-19T15:47:25.476 INFO:tasks.workunit.client.0.smithi102.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 6.7239e-05 s, 22.8 MB/s 2024-08-19T15:47:25.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.470418 2024-08-19T15:47:25.693 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-08-19T15:47:25.693 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-08-19T15:47:25.901 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap7 2024-08-19T15:47:25.903 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-08-19T15:47:26.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-08-19T15:47:26.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-08-19T15:47:26.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-08-19T15:47:26.447 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:47:26.448 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:47:26.448 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:47:26.448 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:47:26.448 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:47:26.753 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:47:26.754 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj1 2024-08-19T15:47:28.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:47:28.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-08-19T15:47:29.046 INFO:tasks.workunit.client.0.smithi102.stdout:WARNING: only removing head with clones present 2024-08-19T15:47:29.046 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:ff7b1f36:::obj1:head# 2024-08-19T15:47:29.562 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-08-19T15:47:29.562 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-08-19T15:47:30.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:47:30.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-08-19T15:47:31.149 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:c52c9666:::obj5:2# 2024-08-19T15:47:31.665 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-08-19T15:47:31.665 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-08-19T15:47:32.633 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:47:32.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:47:32.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-08-19T15:47:33.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-08-19T15:47:33.433 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:47:33.433 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-08-19T15:47:33.433 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:33.433 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-08-19T15:47:33.433 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-08-19T15:47:33.433 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-08-19T15:47:33.433 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-08-19T15:47:33.433 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-08-19T15:47:33.433 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-08-19T15:47:33.434 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-08-19T15:47:33.434 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:47:33.434 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-08-19T15:47:33.434 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-08-19T15:47:33.434 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-08-19T15:47:33.434 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-08-19T15:47:33.434 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:47:33.434 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:47:33.434 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:33.434 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:47:33.434 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:47:33.435 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:47:33.435 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:33.435 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:47:33.435 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:47:33.435 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:47:33.435 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:33.435 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:47:33.435 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:47:33.435 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:47:33.435 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:33.435 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:47:33.435 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:47:33.435 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:47:33.436 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:33.436 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:47:33.436 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:47:33.436 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:47:33.436 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:33.436 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:47:33.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-08-19T15:47:33.437 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-08-19T15:47:33.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-08-19T15:47:34.172 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:c52c9666:::obj5:1# 2024-08-19T15:47:34.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-08-19T15:47:35.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-08-19T15:47:35.741 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:47:35.741 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-08-19T15:47:35.741 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:35.741 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-08-19T15:47:35.741 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-08-19T15:47:35.741 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-08-19T15:47:35.741 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-08-19T15:47:35.741 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-08-19T15:47:35.741 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-08-19T15:47:35.742 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-08-19T15:47:35.742 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:47:35.742 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-08-19T15:47:35.742 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-08-19T15:47:35.742 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-08-19T15:47:35.742 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-08-19T15:47:35.742 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:47:35.742 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:47:35.742 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:35.742 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:47:35.742 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:47:35.743 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:47:35.743 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:35.743 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:47:35.743 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:47:35.743 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:47:35.743 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:35.743 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:47:35.743 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:47:35.743 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:47:35.743 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:35.743 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:47:35.743 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:47:35.743 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:47:35.744 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:35.744 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:47:35.744 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:47:35.744 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:47:35.744 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:35.744 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:47:35.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-08-19T15:47:35.745 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-08-19T15:47:35.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-08-19T15:47:35.745 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-08-19T15:47:35.745 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-08-19T15:47:36.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:47:36.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=18 2024-08-19T15:47:36.551 INFO:tasks.workunit.client.0.smithi102.stderr:18+0 records in 2024-08-19T15:47:36.551 INFO:tasks.workunit.client.0.smithi102.stderr:18+0 records out 2024-08-19T15:47:36.552 INFO:tasks.workunit.client.0.smithi102.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000114809 s, 40.1 MB/s 2024-08-19T15:47:36.552 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.470418 2024-08-19T15:47:37.597 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj3 2024-08-19T15:47:39.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:47:39.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=15 2024-08-19T15:47:39.170 INFO:tasks.workunit.client.0.smithi102.stderr:15+0 records in 2024-08-19T15:47:39.170 INFO:tasks.workunit.client.0.smithi102.stderr:15+0 records out 2024-08-19T15:47:39.170 INFO:tasks.workunit.client.0.smithi102.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000102947 s, 37.3 MB/s 2024-08-19T15:47:39.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.470418 2024-08-19T15:47:40.266 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj4 2024-08-19T15:47:40.266 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-08-19T15:47:41.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:47:41.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-08-19T15:47:41.719 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:0ee9ae15:::obj4:7# 2024-08-19T15:47:42.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-08-19T15:47:43.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-08-19T15:47:43.287 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-08-19T15:47:43.287 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:43.287 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-08-19T15:47:43.287 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-08-19T15:47:43.287 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-08-19T15:47:43.287 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-08-19T15:47:43.287 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-08-19T15:47:43.287 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-08-19T15:47:43.287 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-08-19T15:47:43.287 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:47:43.288 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-08-19T15:47:43.288 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-08-19T15:47:43.288 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-08-19T15:47:43.288 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-08-19T15:47:43.288 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:47:43.288 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:43.288 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:47:43.288 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:47:43.288 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:43.288 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:47:43.288 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:47:43.288 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:43.289 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:47:43.289 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:47:43.289 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:43.289 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:47:43.289 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:47:43.289 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:43.289 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:47:43.289 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:47:43.289 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:43.289 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:47:43.290 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-08-19T15:47:43.290 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:43.290 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:43.290 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:43.290 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:43.290 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:43.290 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:43.290 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:47:43.291 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj16 2024-08-19T15:47:43.291 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-08-19T15:47:44.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:47:44.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-08-19T15:47:44.618 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:461f8b5e:::obj16:7# 2024-08-19T15:47:45.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-08-19T15:47:46.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-08-19T15:47:46.184 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-08-19T15:47:46.184 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-08-19T15:47:46.184 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-08-19T15:47:46.184 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-08-19T15:47:46.184 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-08-19T15:47:46.185 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-08-19T15:47:46.185 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-08-19T15:47:46.185 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-08-19T15:47:46.185 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:47:46.185 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-08-19T15:47:46.185 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-08-19T15:47:46.185 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-08-19T15:47:46.185 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-08-19T15:47:46.185 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:47:46.185 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:47:46.185 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:47:46.185 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:47:46.185 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:47:46.186 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:47:46.186 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:47:46.186 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:47:46.186 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:47:46.186 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:47:46.186 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:47:46.186 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:47:46.186 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-08-19T15:47:46.186 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-08-19T15:47:46.187 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj2 2024-08-19T15:47:47.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:47:47.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-08-19T15:47:48.074 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:47:48.074 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-08-19T15:47:48.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:47:48.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=7 2024-08-19T15:47:48.076 INFO:tasks.workunit.client.0.smithi102.stderr:7+0 records in 2024-08-19T15:47:48.076 INFO:tasks.workunit.client.0.smithi102.stderr:7+0 records out 2024-08-19T15:47:48.076 INFO:tasks.workunit.client.0.smithi102.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 0.000119782 s, 15.0 MB/s 2024-08-19T15:47:48.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.470418 2024-08-19T15:47:49.348 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj6 2024-08-19T15:47:50.136 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:47:50.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:47:50.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-08-19T15:47:51.455 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj7 2024-08-19T15:47:52.257 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:47:53.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:47:53.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-08-19T15:47:54.091 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj8 2024-08-19T15:47:54.737 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:47:55.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:47:55.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-08-19T15:47:56.020 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj9 2024-08-19T15:47:56.691 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:47:57.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:47:57.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-08-19T15:47:58.601 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj10 2024-08-19T15:47:59.262 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:47:59.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:47:59.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-08-19T15:48:00.686 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj11 2024-08-19T15:48:01.332 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:48:01.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:48:01.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-08-19T15:48:02.749 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj12 2024-08-19T15:48:03.427 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:48:03.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:48:03.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-08-19T15:48:04.834 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj13 2024-08-19T15:48:05.447 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:48:05.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:48:05.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-08-19T15:48:06.800 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj14 2024-08-19T15:48:07.495 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:48:08.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:48:08.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-08-19T15:48:09.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-08-19T15:48:09.097 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj15 2024-08-19T15:48:09.714 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:48:09.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:48:09.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-08-19T15:48:11.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-08-19T15:48:11.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-08-19T15:48:11.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: rm -f testdata.470418 2024-08-19T15:48:11.044 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: expr 2 - 1 2024-08-19T15:48:11.045 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: seq 0 1 2024-08-19T15:48:11.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:48:11.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2024-08-19T15:48:11.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-08-19T15:48:11.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:48:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:48:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:48:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-08-19T15:48:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-08-19T15:48:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:48:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:48:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:48:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-08-19T15:48:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-08-19T15:48:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:48:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:48:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-08-19T15:48:11.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:48:11.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:48:11.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:48:11.051 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:48:11.051 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:48:11.051 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:48:11.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:48:11.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:48:11.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:48:11.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-08-19T15:48:11.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-08-19T15:48:11.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:48:11.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:48:11.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:48:11.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:48:11.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:48:11.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:48:11.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-08-19T15:48:11.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:48:11.052 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:48:11.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:48:11.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-08-19T15:48:11.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:48:11.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:48:11.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:48:11.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:48:11.092 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:11.090+0000 7f820b41e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:48:11.093 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:11.092+0000 7f820b41e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:48:11.095 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:11.093+0000 7f820b41e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:48:11.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:48:11.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:48:11.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:48:11.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:48:11.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:48:11.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:48:11.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:48:11.342 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:48:11.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:48:11.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:48:11.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:11.609+0000 7f820b41e640 -1 Falling back to public interface 2024-08-19T15:48:11.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:48:12.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:48:12.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:48:12.665 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:48:12.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:48:12.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:48:12.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:48:12.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:48:13.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:48:13.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:48:13.953 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:48:13.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:48:13.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:48:13.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:48:14.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:48:14.649 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:14.647+0000 7f820b41e640 -1 osd.0 23 log_to_monitors true 2024-08-19T15:48:15.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:48:15.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:48:15.238 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:48:15.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:48:15.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:48:15.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:48:15.549 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 24 last_clean_interval [5,23) [v2:127.0.0.1:6802/2137826146,v1:127.0.0.1:6803/2137826146] [v2:127.0.0.1:6804/2137826146,v1:127.0.0.1:6805/2137826146] exists,up 35afcfe8-0dcc-411c-bb7d-4b4e3b58909d 2024-08-19T15:48:15.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:48:15.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:48:15.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:48:15.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:48:15.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2024-08-19T15:48:15.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-08-19T15:48:15.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:48:15.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:48:15.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:48:15.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/1 2024-08-19T15:48:15.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-08-19T15:48:15.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:48:15.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:48:15.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:48:15.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-08-19T15:48:15.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-08-19T15:48:15.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:48:15.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:48:15.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-08-19T15:48:15.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:48:15.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:48:15.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:48:15.555 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:48:15.555 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:48:15.555 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:48:15.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:48:15.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:48:15.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:48:15.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-08-19T15:48:15.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-08-19T15:48:15.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:48:15.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:48:15.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:48:15.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:48:15.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:48:15.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:48:15.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/1 2024-08-19T15:48:15.556 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:48:15.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:48:15.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:48:15.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/1/whoami 2024-08-19T15:48:15.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:48:15.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:48:15.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:48:15.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:48:15.589 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:15.587+0000 7fd71e744640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:48:15.611 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:15.610+0000 7fd71e744640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:48:15.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:15.611+0000 7fd71e744640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:48:15.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:48:15.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:48:15.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:48:15.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:48:15.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:48:15.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:48:15.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:48:15.871 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:48:15.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:48:15.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:48:16.129 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:16.128+0000 7fd71e744640 -1 Falling back to public interface 2024-08-19T15:48:16.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:48:17.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:48:17.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:48:17.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:48:17.177 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:48:17.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:48:17.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:48:17.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:48:18.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:48:18.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:48:18.466 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:48:18.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:48:18.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:48:18.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:48:18.476 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:18.474+0000 7fd71e744640 -1 osd.1 23 log_to_monitors true 2024-08-19T15:48:18.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:48:19.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:48:19.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:48:19.758 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:48:19.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:48:19.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:48:19.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:48:20.094 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6810/3385701463,v1:127.0.0.1:6811/3385701463] [v2:127.0.0.1:6812/3385701463,v1:127.0.0.1:6813/3385701463] exists,up 6d06ba29-0bfc-49cb-ad80-79014868d11b 2024-08-19T15:48:20.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:48:20.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:48:20.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:48:20.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2024-08-19T15:48:20.201 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: { 2024-08-19T15:48:20.201 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-08-19T15:48:20.201 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:48:20.214 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: { 2024-08-19T15:48:20.214 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-08-19T15:48:20.214 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:48:20.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:775: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 3 2024-08-19T15:48:20.331 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: { 2024-08-19T15:48:20.331 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-08-19T15:48:20.331 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:48:20.345 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: { 2024-08-19T15:48:20.345 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-08-19T15:48:20.345 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:48:20.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_scrub_chunk_min 3 2024-08-19T15:48:20.482 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: { 2024-08-19T15:48:20.482 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-08-19T15:48:20.482 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:48:20.496 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: { 2024-08-19T15:48:20.496 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-08-19T15:48:20.496 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:48:20.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:777: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2024-08-19T15:48:20.624 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: { 2024-08-19T15:48:20.624 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-08-19T15:48:20.624 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:48:20.637 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: { 2024-08-19T15:48:20.638 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-08-19T15:48:20.638 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:48:20.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2024-08-19T15:48:20.758 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: { 2024-08-19T15:48:20.758 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-08-19T15:48:20.758 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:48:20.772 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: { 2024-08-19T15:48:20.772 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-08-19T15:48:20.772 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:48:20.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:779: _scrub_snaps_multi: wait_for_clean 2024-08-19T15:48:20.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:48:20.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:48:20.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:48:20.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:48:20.782 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:48:20.782 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:48:20.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:48:20.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:48:20.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:48:20.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:48:20.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:48:20.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:48:20.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:48:20.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:48:20.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:48:21.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:48:21.149 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:48:21.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:48:21.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:48:21.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:48:21.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149699 2024-08-19T15:48:21.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149699 2024-08-19T15:48:21.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149699' 2024-08-19T15:48:21.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:48:21.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:48:21.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-08-19T15:48:21.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-08-19T15:48:21.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149699 1-124554051586' 2024-08-19T15:48:21.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:48:21.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149699 2024-08-19T15:48:21.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:48:21.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:48:21.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149699 2024-08-19T15:48:21.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:48:21.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149699 2024-08-19T15:48:21.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149699' 2024-08-19T15:48:21.422 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 111669149699 2024-08-19T15:48:21.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:48:21.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149699 2024-08-19T15:48:21.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:48:22.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:48:22.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:48:22.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149699 -lt 111669149699 2024-08-19T15:48:22.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:48:22.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-124554051586 2024-08-19T15:48:22.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:48:22.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:48:22.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-124554051586 2024-08-19T15:48:22.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:48:22.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-08-19T15:48:22.986 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 124554051586 2024-08-19T15:48:22.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-08-19T15:48:22.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:48:23.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-08-19T15:48:23.265 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:48:23.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:48:23.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:48:23.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:48:23.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:48:23.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:48:23.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:48:23.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:48:23.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:48:23.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:48:23.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:48:23.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:48:23.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:48:23.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:48:23.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:48:24.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:48:24.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:48:24.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:48:24.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:781: _scrub_snaps_multi: local pgid=1.0 2024-08-19T15:48:24.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:782: _scrub_snaps_multi: pg_scrub 1.0 2024-08-19T15:48:24.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=1.0 2024-08-19T15:48:24.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 1.0 2024-08-19T15:48:24.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:48:24.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:48:24.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:48:24.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:48:24.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-08-19T15:47:13.694114+0000 2024-08-19T15:48:24.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 1.0 2024-08-19T15:48:24.816 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 1.0 on osd.1 to scrub 2024-08-19T15:48:24.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 1.0 2024-08-19T15:47:13.694114+0000 2024-08-19T15:48:24.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-08-19T15:48:24.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:47:13.694114+0000 2024-08-19T15:48:24.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:48:24.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:48:24.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:48:24.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:48:24.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:48:24.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:48:24.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:48:24.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:48:25.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:47:13.694114+0000 '>' 2024-08-19T15:47:13.694114+0000 2024-08-19T15:48:25.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:48:26.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:48:26.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:48:26.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:48:26.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:48:26.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:48:26.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:48:26.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:48:26.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:47:13.694114+0000 '>' 2024-08-19T15:47:13.694114+0000 2024-08-19T15:48:26.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:48:27.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:48:27.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:48:27.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:48:27.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:48:27.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:48:27.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:48:27.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:48:27.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:47:13.694114+0000 '>' 2024-08-19T15:47:13.694114+0000 2024-08-19T15:48:27.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:48:28.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:48:28.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:48:28.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:48:28.666 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:48:28.666 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:48:28.666 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:48:28.666 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:48:28.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:47:13.694114+0000 '>' 2024-08-19T15:47:13.694114+0000 2024-08-19T15:48:28.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:48:29.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:48:29.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:48:29.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:48:29.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:48:29.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:48:29.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:48:29.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:48:30.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:48:26.010891+0000 '>' 2024-08-19T15:47:13.694114+0000 2024-08-19T15:48:30.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:48:30.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2024-08-19T15:48:30.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: wc -l 2024-08-19T15:48:30.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: test 14 -gt 3 2024-08-19T15:48:30.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:48:30.238 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: wc -l 2024-08-19T15:48:30.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: test 14 -gt 3 2024-08-19T15:48:30.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:789: _scrub_snaps_multi: rados list-inconsistent-pg test 2024-08-19T15:48:30.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2024-08-19T15:48:30.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: test 1 = 1 2024-08-19T15:48:30.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2024-08-19T15:48:30.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: test 1.0 = 1.0 2024-08-19T15:48:30.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:795: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2024-08-19T15:48:30.316 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:48:30.316 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 29, 2024-08-19T15:48:30.316 INFO:tasks.workunit.client.0.smithi102.stdout: "inconsistents": [ 2024-08-19T15:48:30.316 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.316 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:48:30.316 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj5", 2024-08-19T15:48:30.316 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:48:30.316 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:48:30.316 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:48:30.316 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 5 2024-08-19T15:48:30.316 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.317 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.317 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [ 2024-08-19T15:48:30.317 INFO:tasks.workunit.client.0.smithi102.stdout: "missing" 2024-08-19T15:48:30.317 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.317 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:48:30.317 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:48:30.317 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj5", 2024-08-19T15:48:30.317 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:48:30.317 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": 1, 2024-08-19T15:48:30.317 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 1718170787, 2024-08-19T15:48:30.317 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:48:30.317 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:48:30.318 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:48:30.318 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.318 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'19", 2024-08-19T15:48:30.318 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'5", 2024-08-19T15:48:30.318 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4241.0:1", 2024-08-19T15:48:30.318 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 5, 2024-08-19T15:48:30.318 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:48:30.318 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:47:19.257044+0000", 2024-08-19T15:48:30.318 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:47:19.257569+0000", 2024-08-19T15:48:30.318 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:48:30.318 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:48:30.318 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:48:30.318 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:48:30.319 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.319 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:48:30.319 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:48:30.319 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0xe07c11e6", 2024-08-19T15:48:30.319 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:48:30.319 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:48:30.319 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:48:30.319 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:48:30.319 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:48:30.319 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:48:30.319 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.319 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:48:30.319 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.320 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:48:30.320 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.320 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:48:30.320 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:48:30.320 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.320 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032 2024-08-19T15:48:30.320 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.320 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.320 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:48:30.320 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:48:30.320 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.320 INFO:tasks.workunit.client.0.smithi102.stdout: "missing" 2024-08-19T15:48:30.320 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.320 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.321 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.321 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.321 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.321 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:48:30.321 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj5", 2024-08-19T15:48:30.321 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:48:30.321 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:48:30.321 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 2, 2024-08-19T15:48:30.321 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 20 2024-08-19T15:48:30.321 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.321 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.321 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [ 2024-08-19T15:48:30.322 INFO:tasks.workunit.client.0.smithi102.stdout: "missing" 2024-08-19T15:48:30.322 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.322 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:48:30.322 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:48:30.322 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj5", 2024-08-19T15:48:30.322 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:48:30.322 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": 2, 2024-08-19T15:48:30.322 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 1718170787, 2024-08-19T15:48:30.322 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:48:30.322 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:48:30.322 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:48:30.322 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.323 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "18'41", 2024-08-19T15:48:30.323 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "17'20", 2024-08-19T15:48:30.323 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4289.0:1", 2024-08-19T15:48:30.323 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 20, 2024-08-19T15:48:30.323 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.323 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:47:22.212198+0000", 2024-08-19T15:48:30.323 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:47:22.212728+0000", 2024-08-19T15:48:30.323 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:48:30.323 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:48:30.323 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:48:30.323 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:48:30.323 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.324 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:48:30.324 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:48:30.324 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0x398c0d89", 2024-08-19T15:48:30.324 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:48:30.324 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:48:30.324 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:48:30.324 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:48:30.324 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:48:30.324 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:48:30.324 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.324 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:48:30.324 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.324 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:48:30.325 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.325 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:48:30.325 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:48:30.325 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.325 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256 2024-08-19T15:48:30.325 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.325 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.325 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:48:30.325 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:48:30.325 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.325 INFO:tasks.workunit.client.0.smithi102.stdout: "missing" 2024-08-19T15:48:30.325 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.326 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.326 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.326 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.326 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.326 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:48:30.326 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj5", 2024-08-19T15:48:30.326 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:48:30.326 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:48:30.326 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 4, 2024-08-19T15:48:30.326 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 42 2024-08-19T15:48:30.326 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.326 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.327 INFO:tasks.workunit.client.0.smithi102.stdout: "size_mismatch" 2024-08-19T15:48:30.327 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.327 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [ 2024-08-19T15:48:30.327 INFO:tasks.workunit.client.0.smithi102.stdout: "size_mismatch_info", 2024-08-19T15:48:30.327 INFO:tasks.workunit.client.0.smithi102.stdout: "obj_size_info_mismatch" 2024-08-19T15:48:30.327 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.327 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:48:30.327 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:48:30.327 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj5", 2024-08-19T15:48:30.327 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:48:30.327 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": 4, 2024-08-19T15:48:30.327 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 1718170787, 2024-08-19T15:48:30.327 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:48:30.328 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:48:30.328 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:48:30.328 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.328 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "20'45", 2024-08-19T15:48:30.328 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "18'42", 2024-08-19T15:48:30.328 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4325.0:1", 2024-08-19T15:48:30.328 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 42, 2024-08-19T15:48:30.328 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 512, 2024-08-19T15:48:30.328 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:47:24.011513+0000", 2024-08-19T15:48:30.328 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:47:24.032674+0000", 2024-08-19T15:48:30.329 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:48:30.329 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:48:30.329 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:48:30.329 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:48:30.329 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.329 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:48:30.329 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:48:30.329 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0x89b85663", 2024-08-19T15:48:30.329 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:48:30.329 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:48:30.329 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:48:30.329 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:48:30.329 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:48:30.329 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:48:30.330 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.330 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:48:30.330 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.330 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:48:30.330 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.330 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:48:30.330 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:48:30.330 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.330 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 512 2024-08-19T15:48:30.330 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.330 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.330 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:48:30.331 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:48:30.331 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.331 INFO:tasks.workunit.client.0.smithi102.stdout: "size_mismatch_info", 2024-08-19T15:48:30.331 INFO:tasks.workunit.client.0.smithi102.stdout: "obj_size_info_mismatch" 2024-08-19T15:48:30.331 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.331 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 4608, 2024-08-19T15:48:30.331 INFO:tasks.workunit.client.0.smithi102.stdout: "object_info": { 2024-08-19T15:48:30.331 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:48:30.331 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj5", 2024-08-19T15:48:30.331 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:48:30.331 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": 4, 2024-08-19T15:48:30.331 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 1718170787, 2024-08-19T15:48:30.331 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:48:30.332 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:48:30.332 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:48:30.332 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.332 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "20'45", 2024-08-19T15:48:30.332 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "18'42", 2024-08-19T15:48:30.332 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4325.0:1", 2024-08-19T15:48:30.332 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 42, 2024-08-19T15:48:30.332 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 512, 2024-08-19T15:48:30.332 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:47:24.011513+0000", 2024-08-19T15:48:30.332 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:47:24.032674+0000", 2024-08-19T15:48:30.332 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:48:30.332 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:48:30.333 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:48:30.333 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:48:30.333 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.333 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:48:30.333 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:48:30.333 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0x89b85663", 2024-08-19T15:48:30.333 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:48:30.333 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:48:30.333 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:48:30.333 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:48:30.333 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:48:30.334 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:48:30.334 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.334 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:48:30.334 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.334 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.334 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.334 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.334 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.334 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:48:30.334 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj4", 2024-08-19T15:48:30.334 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:48:30.335 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:48:30.335 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 7, 2024-08-19T15:48:30.335 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 4 2024-08-19T15:48:30.335 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.335 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.335 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [ 2024-08-19T15:48:30.335 INFO:tasks.workunit.client.0.smithi102.stdout: "missing" 2024-08-19T15:48:30.335 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.335 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:48:30.335 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:48:30.335 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj4", 2024-08-19T15:48:30.335 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:48:30.335 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": 7, 2024-08-19T15:48:30.336 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 2826278768, 2024-08-19T15:48:30.336 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:48:30.336 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:48:30.336 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:48:30.336 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.336 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "23'51", 2024-08-19T15:48:30.336 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'4", 2024-08-19T15:48:30.336 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4238.0:1", 2024-08-19T15:48:30.336 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 4, 2024-08-19T15:48:30.336 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:48:30.336 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:47:19.130803+0000", 2024-08-19T15:48:30.336 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:47:19.131350+0000", 2024-08-19T15:48:30.336 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:48:30.337 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:48:30.337 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:48:30.337 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:48:30.337 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.337 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:48:30.337 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:48:30.337 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0xe07c11e6", 2024-08-19T15:48:30.337 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:48:30.337 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:48:30.337 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:48:30.337 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:48:30.337 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:48:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:48:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:48:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:48:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:48:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:48:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032 2024-08-19T15:48:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:48:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:48:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: "missing" 2024-08-19T15:48:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:48:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj5", 2024-08-19T15:48:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:48:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:48:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 7, 2024-08-19T15:48:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 0 2024-08-19T15:48:30.340 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.340 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.340 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [ 2024-08-19T15:48:30.340 INFO:tasks.workunit.client.0.smithi102.stdout: "missing", 2024-08-19T15:48:30.340 INFO:tasks.workunit.client.0.smithi102.stdout: "info_missing" 2024-08-19T15:48:30.340 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.340 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:48:30.340 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.340 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:48:30.340 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:48:30.340 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.340 INFO:tasks.workunit.client.0.smithi102.stdout: "missing" 2024-08-19T15:48:30.340 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.340 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.341 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.341 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:48:30.341 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:48:30.341 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.341 INFO:tasks.workunit.client.0.smithi102.stdout: "info_missing" 2024-08-19T15:48:30.341 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.341 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1792 2024-08-19T15:48:30.341 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.341 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.341 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.341 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.341 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:48:30.341 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj1", 2024-08-19T15:48:30.342 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:48:30.342 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:48:30.342 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:48:30.342 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 18 2024-08-19T15:48:30.342 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.342 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.342 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [ 2024-08-19T15:48:30.342 INFO:tasks.workunit.client.0.smithi102.stdout: "missing" 2024-08-19T15:48:30.342 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.342 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:48:30.342 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:48:30.342 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj1", 2024-08-19T15:48:30.342 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:48:30.342 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:48:30.343 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 1828249343, 2024-08-19T15:48:30.343 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:48:30.343 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:48:30.343 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:48:30.343 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.343 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'18", 2024-08-19T15:48:30.343 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'1", 2024-08-19T15:48:30.343 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4286.0:1", 2024-08-19T15:48:30.343 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 18, 2024-08-19T15:48:30.344 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.344 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:47:21.985478+0000", 2024-08-19T15:48:30.344 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:47:22.073053+0000", 2024-08-19T15:48:30.344 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:48:30.344 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:48:30.344 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:48:30.344 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:48:30.344 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.344 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:48:30.344 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:48:30.344 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0x398c0d89", 2024-08-19T15:48:30.344 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:48:30.344 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:48:30.345 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:48:30.345 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:48:30.345 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:48:30.345 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:48:30.345 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.345 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:48:30.345 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.345 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:48:30.345 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.345 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:48:30.345 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:48:30.345 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.345 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256 2024-08-19T15:48:30.345 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.346 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.346 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:48:30.346 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:48:30.346 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.346 INFO:tasks.workunit.client.0.smithi102.stdout: "missing" 2024-08-19T15:48:30.346 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.346 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.346 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.346 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.346 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.346 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:48:30.346 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj10", 2024-08-19T15:48:30.346 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:48:30.347 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:48:30.347 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:48:30.347 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 32 2024-08-19T15:48:30.347 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.347 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.347 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_inconsistency" 2024-08-19T15:48:30.347 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.347 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [], 2024-08-19T15:48:30.347 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:48:30.347 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:48:30.347 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj10", 2024-08-19T15:48:30.347 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:48:30.347 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:48:30.348 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 718195851, 2024-08-19T15:48:30.348 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:48:30.348 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:48:30.348 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:48:30.348 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.348 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'32", 2024-08-19T15:48:30.348 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'10", 2024-08-19T15:48:30.348 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4307.0:1", 2024-08-19T15:48:30.348 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 32, 2024-08-19T15:48:30.348 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.348 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:47:23.115500+0000", 2024-08-19T15:48:30.348 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:47:23.116035+0000", 2024-08-19T15:48:30.348 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:48:30.348 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:48:30.349 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:48:30.349 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:48:30.349 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.349 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:48:30.349 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:48:30.349 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0x398c0d89", 2024-08-19T15:48:30.349 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:48:30.349 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:48:30.349 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:48:30.349 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:48:30.350 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:48:30.350 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:48:30.350 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.350 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:48:30.350 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.350 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:48:30.350 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.350 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:48:30.350 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:48:30.350 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.350 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.350 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:48:30.350 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:48:30.351 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:48:30.351 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.351 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:48:30.351 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:48:30.351 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:48:30.351 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:48:30.351 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:48:30.351 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.351 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.351 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.351 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.352 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.352 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.352 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:48:30.352 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:48:30.352 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.352 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.352 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:48:30.352 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:48:30.352 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:48:30.352 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.352 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:48:30.353 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:48:30.353 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "????", 2024-08-19T15:48:30.353 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:48:30.353 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:48:30.353 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.353 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.353 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.353 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.353 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.353 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.353 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.353 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.353 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:48:30.353 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj11", 2024-08-19T15:48:30.354 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:48:30.354 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:48:30.354 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:48:30.354 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 34 2024-08-19T15:48:30.354 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.354 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.354 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_inconsistency" 2024-08-19T15:48:30.354 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.354 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [], 2024-08-19T15:48:30.354 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:48:30.354 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:48:30.354 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj11", 2024-08-19T15:48:30.354 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:48:30.355 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:48:30.355 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 693400951, 2024-08-19T15:48:30.355 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:48:30.355 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:48:30.355 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:48:30.355 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.355 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'34", 2024-08-19T15:48:30.355 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'11", 2024-08-19T15:48:30.355 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4310.0:1", 2024-08-19T15:48:30.355 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 34, 2024-08-19T15:48:30.355 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.355 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:47:23.249249+0000", 2024-08-19T15:48:30.355 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:47:23.249832+0000", 2024-08-19T15:48:30.356 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:48:30.356 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:48:30.356 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:48:30.356 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:48:30.356 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.356 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:48:30.356 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:48:30.356 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0x398c0d89", 2024-08-19T15:48:30.356 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:48:30.356 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:48:30.356 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:48:30.356 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:48:30.356 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:48:30.356 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:48:30.357 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.357 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:48:30.357 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.357 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:48:30.357 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.357 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:48:30.357 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:48:30.357 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.357 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.357 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:48:30.357 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:48:30.357 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:48:30.357 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.358 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:48:30.358 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:48:30.358 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:48:30.358 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:48:30.358 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:48:30.358 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.358 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.358 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.358 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.358 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.358 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.358 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:48:30.359 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:48:30.359 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.359 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.359 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:48:30.359 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:48:30.359 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [] 2024-08-19T15:48:30.359 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.359 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.359 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.359 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.359 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.359 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:48:30.360 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj13", 2024-08-19T15:48:30.360 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:48:30.360 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:48:30.360 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:48:30.360 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 38 2024-08-19T15:48:30.360 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.360 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.360 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_inconsistency" 2024-08-19T15:48:30.360 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.360 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [], 2024-08-19T15:48:30.360 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:48:30.360 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:48:30.361 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj13", 2024-08-19T15:48:30.361 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:48:30.361 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:48:30.361 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 2087409765, 2024-08-19T15:48:30.361 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:48:30.361 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:48:30.361 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:48:30.361 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.361 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'38", 2024-08-19T15:48:30.361 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'13", 2024-08-19T15:48:30.361 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4316.0:1", 2024-08-19T15:48:30.361 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 38, 2024-08-19T15:48:30.362 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.362 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:47:23.517173+0000", 2024-08-19T15:48:30.362 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:47:23.517694+0000", 2024-08-19T15:48:30.362 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:48:30.362 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:48:30.362 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:48:30.362 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:48:30.362 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.362 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:48:30.362 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:48:30.362 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0x398c0d89", 2024-08-19T15:48:30.362 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:48:30.362 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:48:30.363 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:48:30.363 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:48:30.363 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:48:30.363 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:48:30.363 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.363 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:48:30.363 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.363 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:48:30.363 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.363 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:48:30.363 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:48:30.363 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.363 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.363 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:48:30.364 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:48:30.364 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:48:30.364 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.364 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:48:30.364 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:48:30.364 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:48:30.364 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:48:30.364 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:48:30.364 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.364 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.364 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.364 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.364 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.365 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.365 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:48:30.365 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:48:30.365 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.365 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.365 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:48:30.365 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:48:30.365 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:48:30.365 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.365 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:48:30.365 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:48:30.365 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]" 2024-08-19T15:48:30.365 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.366 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.366 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.366 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.366 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.366 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.366 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.366 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:48:30.366 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj14", 2024-08-19T15:48:30.366 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:48:30.366 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:48:30.366 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:48:30.366 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 40 2024-08-19T15:48:30.366 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.367 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.367 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_inconsistency" 2024-08-19T15:48:30.367 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.367 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [], 2024-08-19T15:48:30.367 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:48:30.367 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:48:30.367 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj14", 2024-08-19T15:48:30.367 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:48:30.367 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:48:30.367 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 2484217095, 2024-08-19T15:48:30.367 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:48:30.367 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:48:30.367 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:48:30.368 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.368 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'40", 2024-08-19T15:48:30.368 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'14", 2024-08-19T15:48:30.368 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4319.0:1", 2024-08-19T15:48:30.368 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 40, 2024-08-19T15:48:30.368 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.368 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:47:23.659211+0000", 2024-08-19T15:48:30.368 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:47:23.659765+0000", 2024-08-19T15:48:30.368 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:48:30.368 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:48:30.368 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:48:30.368 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:48:30.368 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.368 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:48:30.369 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:48:30.369 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0x398c0d89", 2024-08-19T15:48:30.369 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:48:30.369 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:48:30.369 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:48:30.369 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:48:30.369 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:48:30.369 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:48:30.369 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.369 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:48:30.369 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.369 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:48:30.369 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.369 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:48:30.370 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:48:30.370 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.370 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.370 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:48:30.370 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:48:30.370 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:48:30.370 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.370 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:48:30.370 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:48:30.370 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:48:30.370 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:48:30.370 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:48:30.370 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.371 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.371 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.371 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.371 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.371 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.371 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:48:30.371 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:48:30.371 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.371 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.371 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:48:30.371 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:48:30.371 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:48:30.371 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.371 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:48:30.372 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1033, 2024-08-19T15:48:30.372 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:48:30.372 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:48:30.372 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:48:30.372 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.372 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.372 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.372 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.372 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.372 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.372 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.373 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.373 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:48:30.373 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj15", 2024-08-19T15:48:30.373 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:48:30.373 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:48:30.373 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:48:30.373 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 15 2024-08-19T15:48:30.373 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.373 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.373 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [ 2024-08-19T15:48:30.373 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_corrupted" 2024-08-19T15:48:30.373 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.374 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:48:30.374 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:48:30.374 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj15", 2024-08-19T15:48:30.374 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:48:30.374 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:48:30.374 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 612772309, 2024-08-19T15:48:30.374 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:48:30.374 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:48:30.374 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:48:30.374 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.374 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "16'15", 2024-08-19T15:48:30.374 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "0'0", 2024-08-19T15:48:30.375 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4271.0:1", 2024-08-19T15:48:30.375 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 15, 2024-08-19T15:48:30.375 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:48:30.375 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:47:20.630678+0000", 2024-08-19T15:48:30.375 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:47:20.631304+0000", 2024-08-19T15:48:30.375 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:48:30.375 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:48:30.375 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:48:30.375 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:48:30.375 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.375 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:48:30.375 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:48:30.375 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0xe07c11e6", 2024-08-19T15:48:30.375 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:48:30.376 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:48:30.376 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:48:30.376 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:48:30.376 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:48:30.376 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:48:30.376 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.376 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:48:30.376 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.376 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:48:30.376 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.376 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:48:30.376 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:48:30.376 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.376 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:48:30.377 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:48:30.377 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-08-19T15:48:30.377 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [] 2024-08-19T15:48:30.377 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.377 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.377 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.377 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:48:30.377 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:48:30.377 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.377 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_corrupted" 2024-08-19T15:48:30.377 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.377 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:48:30.377 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": "Z2FyYmFnZQo=" 2024-08-19T15:48:30.377 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.378 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.378 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.378 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.378 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:48:30.378 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj2", 2024-08-19T15:48:30.378 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:48:30.378 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:48:30.378 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:48:30.378 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 56 2024-08-19T15:48:30.378 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.378 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.378 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [ 2024-08-19T15:48:30.378 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_missing" 2024-08-19T15:48:30.379 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.379 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:48:30.379 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:48:30.379 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj2", 2024-08-19T15:48:30.379 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:48:30.379 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:48:30.379 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 1058988552, 2024-08-19T15:48:30.379 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:48:30.379 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:48:30.379 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:48:30.379 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.379 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "23'56", 2024-08-19T15:48:30.379 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "20'48", 2024-08-19T15:48:30.379 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4361.0:1", 2024-08-19T15:48:30.380 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 56, 2024-08-19T15:48:30.380 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 0, 2024-08-19T15:48:30.380 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:47:26.330013+0000", 2024-08-19T15:48:30.380 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:47:26.330597+0000", 2024-08-19T15:48:30.380 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:48:30.380 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:48:30.380 INFO:tasks.workunit.client.0.smithi102.stdout: "whiteout", 2024-08-19T15:48:30.380 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty" 2024-08-19T15:48:30.380 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.380 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:48:30.380 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:48:30.380 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0xffffffff", 2024-08-19T15:48:30.380 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:48:30.381 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:48:30.381 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:48:30.381 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:48:30.381 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:48:30.381 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:48:30.381 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.381 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:48:30.381 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.381 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:48:30.381 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.382 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:48:30.382 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:48:30.382 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.382 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 0, 2024-08-19T15:48:30.382 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:48:30.382 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 7, 2024-08-19T15:48:30.382 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:48:30.382 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.382 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 4, 2024-08-19T15:48:30.382 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:48:30.382 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:48:30.382 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:48:30.382 INFO:tasks.workunit.client.0.smithi102.stdout: 4, 2024-08-19T15:48:30.382 INFO:tasks.workunit.client.0.smithi102.stdout: 3, 2024-08-19T15:48:30.383 INFO:tasks.workunit.client.0.smithi102.stdout: 2, 2024-08-19T15:48:30.383 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:48:30.383 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.383 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.383 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.383 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 7, 2024-08-19T15:48:30.383 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1024, 2024-08-19T15:48:30.383 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:48:30.383 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:48:30.383 INFO:tasks.workunit.client.0.smithi102.stdout: 7, 2024-08-19T15:48:30.383 INFO:tasks.workunit.client.0.smithi102.stdout: 6, 2024-08-19T15:48:30.383 INFO:tasks.workunit.client.0.smithi102.stdout: 5 2024-08-19T15:48:30.383 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.384 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.384 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.384 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.384 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.384 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.384 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:48:30.384 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:48:30.384 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.384 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_missing" 2024-08-19T15:48:30.384 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.384 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 0 2024-08-19T15:48:30.384 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.384 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.384 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.385 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.385 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:48:30.385 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj3", 2024-08-19T15:48:30.385 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:48:30.385 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:48:30.385 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:48:30.385 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 44 2024-08-19T15:48:30.385 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.385 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.385 INFO:tasks.workunit.client.0.smithi102.stdout: "size_mismatch" 2024-08-19T15:48:30.385 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.385 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [ 2024-08-19T15:48:30.385 INFO:tasks.workunit.client.0.smithi102.stdout: "size_mismatch_info", 2024-08-19T15:48:30.385 INFO:tasks.workunit.client.0.smithi102.stdout: "obj_size_info_mismatch" 2024-08-19T15:48:30.386 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.386 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:48:30.386 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:48:30.386 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj3", 2024-08-19T15:48:30.386 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:48:30.386 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:48:30.386 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 1643547569, 2024-08-19T15:48:30.386 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:48:30.386 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:48:30.386 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:48:30.386 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.386 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "19'44", 2024-08-19T15:48:30.386 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "17'22", 2024-08-19T15:48:30.387 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4331.0:1", 2024-08-19T15:48:30.387 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 44, 2024-08-19T15:48:30.387 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 768, 2024-08-19T15:48:30.387 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:47:24.415716+0000", 2024-08-19T15:48:30.387 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:47:24.434283+0000", 2024-08-19T15:48:30.387 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:48:30.387 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:48:30.387 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:48:30.387 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:48:30.387 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.387 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:48:30.387 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:48:30.387 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0x517b6a68", 2024-08-19T15:48:30.388 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:48:30.388 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:48:30.388 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:48:30.388 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:48:30.388 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:48:30.388 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:48:30.388 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.388 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:48:30.388 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.388 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:48:30.388 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.388 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:48:30.388 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:48:30.389 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.389 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 768 2024-08-19T15:48:30.389 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.389 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.389 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:48:30.389 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:48:30.389 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.389 INFO:tasks.workunit.client.0.smithi102.stdout: "size_mismatch_info", 2024-08-19T15:48:30.389 INFO:tasks.workunit.client.0.smithi102.stdout: "obj_size_info_mismatch" 2024-08-19T15:48:30.389 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.389 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 3840, 2024-08-19T15:48:30.389 INFO:tasks.workunit.client.0.smithi102.stdout: "object_info": { 2024-08-19T15:48:30.389 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:48:30.390 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj3", 2024-08-19T15:48:30.390 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:48:30.390 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:48:30.390 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 1643547569, 2024-08-19T15:48:30.390 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:48:30.390 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:48:30.390 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:48:30.390 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.390 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "19'44", 2024-08-19T15:48:30.390 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "17'22", 2024-08-19T15:48:30.390 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4331.0:1", 2024-08-19T15:48:30.390 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 44, 2024-08-19T15:48:30.390 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 768, 2024-08-19T15:48:30.391 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:47:24.415716+0000", 2024-08-19T15:48:30.391 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:47:24.434283+0000", 2024-08-19T15:48:30.391 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:48:30.391 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:48:30.391 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:48:30.391 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:48:30.391 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.391 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:48:30.391 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:48:30.391 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0x517b6a68", 2024-08-19T15:48:30.391 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:48:30.391 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:48:30.391 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:48:30.391 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:48:30.392 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:48:30.392 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:48:30.392 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.392 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:48:30.392 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.392 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.392 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.392 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.392 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.392 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:48:30.392 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj6", 2024-08-19T15:48:30.392 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:48:30.392 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:48:30.392 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:48:30.393 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 24 2024-08-19T15:48:30.393 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.393 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.393 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_inconsistency" 2024-08-19T15:48:30.393 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.393 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [], 2024-08-19T15:48:30.393 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:48:30.393 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:48:30.393 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj6", 2024-08-19T15:48:30.393 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:48:30.393 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:48:30.393 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 2202164420, 2024-08-19T15:48:30.393 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:48:30.394 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:48:30.394 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:48:30.394 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.394 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'24", 2024-08-19T15:48:30.394 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'6", 2024-08-19T15:48:30.394 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4295.0:1", 2024-08-19T15:48:30.394 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 24, 2024-08-19T15:48:30.394 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.394 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:47:22.581079+0000", 2024-08-19T15:48:30.394 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:47:22.581633+0000", 2024-08-19T15:48:30.394 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:48:30.394 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:48:30.394 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:48:30.394 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:48:30.395 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.395 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:48:30.395 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:48:30.395 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0x398c0d89", 2024-08-19T15:48:30.395 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:48:30.395 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:48:30.395 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:48:30.395 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:48:30.395 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:48:30.395 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:48:30.395 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.395 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:48:30.395 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.396 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:48:30.396 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.396 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:48:30.396 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:48:30.396 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.396 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.396 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:48:30.396 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:48:30.396 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:48:30.396 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.396 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:48:30.396 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:48:30.396 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:48:30.397 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:48:30.397 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:48:30.397 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.397 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.397 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.397 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.397 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.397 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.397 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:48:30.397 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:48:30.397 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.397 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.398 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:48:30.398 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:48:30.398 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [] 2024-08-19T15:48:30.398 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.398 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.398 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.398 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.398 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.398 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:48:30.398 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj7", 2024-08-19T15:48:30.398 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:48:30.398 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:48:30.399 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:48:30.399 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 26 2024-08-19T15:48:30.399 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.399 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.399 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_inconsistency" 2024-08-19T15:48:30.399 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.399 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [], 2024-08-19T15:48:30.399 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:48:30.399 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:48:30.399 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj7", 2024-08-19T15:48:30.399 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:48:30.399 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:48:30.399 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 1552453721, 2024-08-19T15:48:30.400 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:48:30.400 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:48:30.400 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:48:30.400 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.400 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'26", 2024-08-19T15:48:30.400 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'7", 2024-08-19T15:48:30.400 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4298.0:1", 2024-08-19T15:48:30.400 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 26, 2024-08-19T15:48:30.400 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.400 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:47:22.721660+0000", 2024-08-19T15:48:30.400 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:47:22.722201+0000", 2024-08-19T15:48:30.400 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:48:30.400 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:48:30.401 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:48:30.401 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:48:30.401 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.401 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:48:30.401 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:48:30.401 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0x398c0d89", 2024-08-19T15:48:30.401 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:48:30.401 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:48:30.401 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:48:30.401 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:48:30.401 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:48:30.401 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:48:30.401 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.402 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:48:30.402 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.402 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:48:30.402 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.402 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:48:30.402 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:48:30.402 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.402 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.402 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:48:30.402 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:48:30.402 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:48:30.402 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.402 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:48:30.403 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:48:30.403 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:48:30.403 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:48:30.403 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:48:30.403 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.403 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.403 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.403 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.403 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.403 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.403 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:48:30.403 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:48:30.403 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.404 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.404 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:48:30.404 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-08-19T15:48:30.404 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [] 2024-08-19T15:48:30.404 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.404 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.404 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.404 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.404 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.404 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:48:30.404 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj8", 2024-08-19T15:48:30.404 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:48:30.404 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:48:30.405 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:48:30.405 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 28 2024-08-19T15:48:30.405 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.405 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.405 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_inconsistency" 2024-08-19T15:48:30.405 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.405 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [], 2024-08-19T15:48:30.405 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:48:30.405 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:48:30.405 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj8", 2024-08-19T15:48:30.405 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:48:30.405 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:48:30.405 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 2381834917, 2024-08-19T15:48:30.406 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:48:30.406 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:48:30.406 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:48:30.406 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.406 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'28", 2024-08-19T15:48:30.406 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'8", 2024-08-19T15:48:30.406 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4301.0:1", 2024-08-19T15:48:30.406 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 28, 2024-08-19T15:48:30.406 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.406 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:47:22.855565+0000", 2024-08-19T15:48:30.406 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:47:22.856116+0000", 2024-08-19T15:48:30.406 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:48:30.406 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:48:30.407 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:48:30.407 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:48:30.407 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.407 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:48:30.407 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:48:30.407 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0x398c0d89", 2024-08-19T15:48:30.407 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:48:30.407 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:48:30.407 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:48:30.407 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:48:30.407 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:48:30.407 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:48:30.407 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.408 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:48:30.408 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.408 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:48:30.408 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.408 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:48:30.408 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:48:30.408 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.408 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.408 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:48:30.408 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:48:30.408 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:48:30.408 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.408 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:48:30.409 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:48:30.409 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:48:30.409 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:48:30.409 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:48:30.409 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.409 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.409 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.409 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.409 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.409 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.409 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:48:30.409 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:48:30.409 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.410 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.410 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:48:30.410 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-08-19T15:48:30.410 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:48:30.410 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.410 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:48:30.410 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:48:30.410 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:48:30.410 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:48:30.410 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:48:30.410 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.410 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.410 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.411 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.411 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.411 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.411 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.411 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.411 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:48:30.411 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj9", 2024-08-19T15:48:30.411 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:48:30.411 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:48:30.411 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:48:30.411 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 30 2024-08-19T15:48:30.411 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.411 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:48:30.411 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_inconsistency" 2024-08-19T15:48:30.412 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.412 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [], 2024-08-19T15:48:30.412 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:48:30.412 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:48:30.412 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj9", 2024-08-19T15:48:30.412 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:48:30.412 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:48:30.412 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 3833113727, 2024-08-19T15:48:30.412 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:48:30.412 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:48:30.412 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:48:30.412 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.412 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'30", 2024-08-19T15:48:30.413 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'9", 2024-08-19T15:48:30.413 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4304.0:1", 2024-08-19T15:48:30.413 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 30, 2024-08-19T15:48:30.413 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.413 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:47:22.989417+0000", 2024-08-19T15:48:30.413 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:47:22.989995+0000", 2024-08-19T15:48:30.413 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:48:30.413 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:48:30.413 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:48:30.413 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:48:30.413 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:48:30.413 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:48:30.413 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:48:30.413 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0x398c0d89", 2024-08-19T15:48:30.414 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:48:30.414 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:48:30.414 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:48:30.414 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:48:30.414 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:48:30.414 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:48:30.414 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.414 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:48:30.414 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.414 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:48:30.414 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.414 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:48:30.414 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:48:30.414 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.415 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.415 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:48:30.415 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:48:30.415 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:48:30.415 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.415 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:48:30.415 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:48:30.415 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:48:30.415 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:48:30.415 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:48:30.415 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.415 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.415 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.416 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.416 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:48:30.416 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.416 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:48:30.416 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:48:30.416 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:48:30.416 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:48:30.416 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:48:30.416 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:48:30.416 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:48:30.416 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:48:30.416 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:48:30.416 INFO:tasks.workunit.client.0.smithi102.stdout: "size": "????", 2024-08-19T15:48:30.416 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:48:30.417 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:48:30.417 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:48:30.417 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.417 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.417 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.417 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.417 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.417 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.417 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:48:30.417 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:48:30.417 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:48:30.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:797: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2024-08-19T15:48:30.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:800: _scrub_snaps_multi: '[' primary = replica ']' 2024-08-19T15:48:30.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:811: _scrub_snaps_multi: scruberrors=30 2024-08-19T15:48:30.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:812: _scrub_snaps_multi: jq .inconsistents 2024-08-19T15:48:30.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:812: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-08-19T15:48:30.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2024-08-19T15:48:30.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-08-19T15:48:30.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1067: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-08-19T15:48:30.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-08-19T15:48:30.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1068: _scrub_snaps_multi: test no = yes 2024-08-19T15:48:30.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1073: _scrub_snaps_multi: test '' = yes 2024-08-19T15:48:30.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: find td/osd-scrub-snaps 2024-08-19T15:48:30.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2024-08-19T15:48:30.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.1.pid 2024-08-19T15:48:30.421 INFO:tasks.workunit.client.0.smithi102.stderr:td/osd-scrub-snaps/osd.0.pid' 2024-08-19T15:48:30.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids= 2024-08-19T15:48:30.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-08-19T15:48:30.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2024-08-19T15:48:30.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='512134 ' 2024-08-19T15:48:30.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-08-19T15:48:30.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2024-08-19T15:48:30.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='511570 ' 2024-08-19T15:48:30.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1085: _scrub_snaps_multi: ERRORS=0 2024-08-19T15:48:30.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1089: _scrub_snaps_multi: '[' primary = primary ']' 2024-08-19T15:48:30.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: seq 1 7 2024-08-19T15:48:30.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-08-19T15:48:30.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap1 2024-08-19T15:48:30.543 INFO:tasks.workunit.client.0.smithi102.stdout:removed pool test snap snap1 2024-08-19T15:48:30.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-08-19T15:48:30.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap2 2024-08-19T15:48:30.793 INFO:tasks.workunit.client.0.smithi102.stdout:removed pool test snap snap2 2024-08-19T15:48:30.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-08-19T15:48:30.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap3 2024-08-19T15:48:31.077 INFO:tasks.workunit.client.0.smithi102.stdout:removed pool test snap snap3 2024-08-19T15:48:31.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-08-19T15:48:31.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap4 2024-08-19T15:48:31.261 INFO:tasks.workunit.client.0.smithi102.stdout:removed pool test snap snap4 2024-08-19T15:48:31.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-08-19T15:48:31.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap5 2024-08-19T15:48:31.504 INFO:tasks.workunit.client.0.smithi102.stdout:removed pool test snap snap5 2024-08-19T15:48:31.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-08-19T15:48:31.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap6 2024-08-19T15:48:31.713 INFO:tasks.workunit.client.0.smithi102.stdout:removed pool test snap snap6 2024-08-19T15:48:31.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-08-19T15:48:31.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap7 2024-08-19T15:48:31.914 INFO:tasks.workunit.client.0.smithi102.stdout:removed pool test snap snap7 2024-08-19T15:48:31.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1095: _scrub_snaps_multi: sleep 5 2024-08-19T15:48:36.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1096: _scrub_snaps_multi: local -i loop=0 2024-08-19T15:48:36.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1097: _scrub_snaps_multi: ceph pg dump pgs 2024-08-19T15:48:36.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1097: _scrub_snaps_multi: grep -q snaptrim 2024-08-19T15:48:37.190 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:48:37.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1099: _scrub_snaps_multi: ceph pg dump pgs 2024-08-19T15:48:37.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1099: _scrub_snaps_multi: grep -q snaptrim_error 2024-08-19T15:48:37.460 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:48:37.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1101: _scrub_snaps_multi: break 2024-08-19T15:48:37.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1111: _scrub_snaps_multi: ceph pg dump pgs 2024-08-19T15:48:37.733 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T15:48:37.733 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent+snaptrim_error 2024-08-19T15:48:31.939745+0000 23'56 37:131 [1,0] 1 [1,0] 1 23'56 2024-08-19T15:48:26.010891+0000 0'0 2024-08-19T15:47:13.694114+0000 7 1 periodic scrub scheduled @ 2024-08-21T01:46:50.133261+0000 33 0 2024-08-19T15:48:37.733 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T15:48:37.733 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T15:48:37.734 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:48:37.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2024-08-19T15:48:37.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 512134 2024-08-19T15:48:37.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2024-08-19T15:48:37.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 511570 2024-08-19T15:48:37.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1122: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2024-08-19T15:48:37.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:48:37.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:48:37.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:48:37.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:48:37.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:48:43.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:48:43.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1124: _scrub_snaps_multi: declare -a err_strings 2024-08-19T15:48:43.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1125: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2024-08-19T15:48:43.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1126: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2024-08-19T15:48:43.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2024-08-19T15:48:43.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2024-08-19T15:48:43.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2024-08-19T15:48:43.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1130: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2024-08-19T15:48:43.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2024-08-19T15:48:43.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1132: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' 2024-08-19T15:48:43.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-08-19T15:48:43.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2024-08-19T15:48:43.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-08-19T15:48:43.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-08-19T15:48:43.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-08-19T15:48:43.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2024-08-19T15:48:43.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-08-19T15:48:43.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2024-08-19T15:48:43.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-08-19T15:48:43.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-08-19T15:48:43.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-08-19T15:48:43.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2024-08-19T15:48:43.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-08-19T15:48:43.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2024-08-19T15:48:43.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-08-19T15:48:43.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' td/osd-scrub-snaps/osd.1.log 2024-08-19T15:48:43.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1144: _scrub_snaps_multi: declare -a rep_err_strings 2024-08-19T15:48:43.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: eval echo '$primary' 2024-08-19T15:48:43.080 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: echo 1 2024-08-19T15:48:43.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: osd=1 2024-08-19T15:48:43.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-08-19T15:48:43.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1147: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2024-08-19T15:48:43.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1149: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.1.log 2024-08-19T15:48:43.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1156: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2024-08-19T15:48:43.083 INFO:tasks.workunit.client.0.smithi102.stdout:TEST PASSED 2024-08-19T15:48:43.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1162: _scrub_snaps_multi: echo 'TEST PASSED' 2024-08-19T15:48:43.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1163: _scrub_snaps_multi: return 0 2024-08-19T15:48:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1181: TEST_scrub_snaps_primary: err=0 2024-08-19T15:48:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1182: TEST_scrub_snaps_primary: CEPH_ARGS='--fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-08-19T15:48:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1183: TEST_scrub_snaps_primary: return 0 2024-08-19T15:48:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-08-19T15:48:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-08-19T15:48:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:48:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-08-19T15:48:43.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:48:43.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:48:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:48:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:48:43.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:48:43.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:48:43.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:48:43.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:48:43.091 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:48:43.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:48:43.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:48:43.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:48:43.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:48:43.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:48:43.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:48:43.095 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:48:43.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:48:43.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:48:43.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-08-19T15:48:43.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:48:43.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:48:43.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:48:43.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.470418 2024-08-19T15:48:43.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:48:43.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:48:43.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-08-19T15:48:43.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-08-19T15:48:43.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-snaps 2024-08-19T15:48:43.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-snaps 2024-08-19T15:48:43.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-08-19T15:48:43.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:48:43.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-08-19T15:48:43.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:48:43.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:48:43.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:48:43.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:48:43.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:48:43.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:48:43.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:48:43.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:48:43.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:48:43.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:48:43.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:48:43.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:48:43.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:48:43.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:48:43.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:48:43.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:48:43.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:48:43.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:48:43.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-08-19T15:48:43.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:48:43.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:48:43.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:48:43.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.470418 2024-08-19T15:48:43.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:48:43.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:48:43.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-snaps 2024-08-19T15:48:43.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T15:48:43.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:48:43.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:48:43.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.470418 2024-08-19T15:48:43.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T15:48:43.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T15:48:43.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T15:48:43.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-08-19T15:48:43.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_replica td/osd-scrub-snaps 2024-08-19T15:48:43.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1167: TEST_scrub_snaps_replica: local dir=td/osd-scrub-snaps 2024-08-19T15:48:43.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1168: TEST_scrub_snaps_replica: ORIG_ARGS='--fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-08-19T15:48:43.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1169: TEST_scrub_snaps_replica: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-08-19T15:48:43.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1170: TEST_scrub_snaps_replica: _scrub_snaps_multi td/osd-scrub-snaps replica 2024-08-19T15:48:43.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2024-08-19T15:48:43.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:733: _scrub_snaps_multi: local poolname=test 2024-08-19T15:48:43.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:734: _scrub_snaps_multi: local OBJS=16 2024-08-19T15:48:43.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:735: _scrub_snaps_multi: local OSDS=2 2024-08-19T15:48:43.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: local which=replica 2024-08-19T15:48:43.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:738: _scrub_snaps_multi: TESTDATA=testdata.470418 2024-08-19T15:48:43.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2024-08-19T15:48:43.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-snaps 2024-08-19T15:48:43.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T15:48:43.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T15:48:43.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T15:48:43.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-snaps/a 2024-08-19T15:48:43.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2024-08-19T15:48:43.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T15:48:43.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:48:43.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:48:43.528 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:48:43.528 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:48:43.528 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:48:43.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:48:43.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-08-19T15:48:43.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T15:48:43.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T15:48:43.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:48:43.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:48:43.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T15:48:43.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T15:48:43.730 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:48:43.730 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:48:43.730 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:48:43.730 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:48:43.730 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:48:43.730 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:48:43.730 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.470418/ceph-mon.a.asok 2024-08-19T15:48:43.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:48:43.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.470418/ceph-mon.a.asok config get fsid 2024-08-19T15:48:43.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T15:48:43.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:48:43.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:48:43.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T15:48:43.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T15:48:43.812 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:48:43.812 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:48:43.812 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:48:43.813 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:48:43.813 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:48:43.813 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:48:43.813 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.470418/ceph-mon.a.asok 2024-08-19T15:48:43.813 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:48:43.813 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.470418/ceph-mon.a.asok config get mon_host 2024-08-19T15:48:43.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:741: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2024-08-19T15:48:43.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-snaps 2024-08-19T15:48:43.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T15:48:43.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T15:48:43.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T15:48:43.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-snaps/x 2024-08-19T15:48:43.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T15:48:44.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T15:48:44.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:48:44.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:48:44.712 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:48:44.712 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:48:44.712 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:48:44.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:48:44.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:48:44.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:48:44.748 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: expr 2 - 1 2024-08-19T15:48:44.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: seq 0 1 2024-08-19T15:48:44.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:48:44.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2024-08-19T15:48:44.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-08-19T15:48:44.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:48:44.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T15:48:44.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:48:44.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-08-19T15:48:44.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-08-19T15:48:44.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:48:44.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:48:44.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:48:44.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-08-19T15:48:44.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-08-19T15:48:44.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:48:44.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:48:44.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-08-19T15:48:44.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:48:44.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:48:44.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:48:44.755 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:48:44.755 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:48:44.755 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:48:44.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:48:44.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:48:44.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:48:44.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:48:44.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:48:44.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-08-19T15:48:44.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-08-19T15:48:44.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:48:44.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:48:44.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:48:44.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:48:44.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:48:44.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:48:44.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-08-19T15:48:44.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:48:44.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d3244e31-0aac-4fc0-aed6-bf16f2fc504b 2024-08-19T15:48:44.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d3244e31-0aac-4fc0-aed6-bf16f2fc504b' 2024-08-19T15:48:44.757 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 d3244e31-0aac-4fc0-aed6-bf16f2fc504b 2024-08-19T15:48:44.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:48:44.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBcacNm7PRrLhAA05qLnlg7Zc1Izl4k6vNqPA== 2024-08-19T15:48:44.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBcacNm7PRrLhAA05qLnlg7Zc1Izl4k6vNqPA=="}' 2024-08-19T15:48:44.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d3244e31-0aac-4fc0-aed6-bf16f2fc504b -i td/osd-scrub-snaps/0/new.json 2024-08-19T15:48:45.002 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:48:45.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-08-19T15:48:45.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBcacNm7PRrLhAA05qLnlg7Zc1Izl4k6vNqPA== --osd-uuid d3244e31-0aac-4fc0-aed6-bf16f2fc504b 2024-08-19T15:48:45.052 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:45.050+0000 7fd2f8957640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:48:45.054 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:45.052+0000 7fd2f8957640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:48:45.055 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:45.053+0000 7fd2f8957640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:48:45.055 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:45.053+0000 7fd2f8957640 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-08-19T15:48:45.055 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:45.053+0000 7fd2f8957640 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-08-19T15:48:45.055 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:45.053+0000 7fd2f8957640 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-08-19T15:48:45.154 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:45.151+0000 7fd2f8957640 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-08-19T15:48:50.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-08-19T15:48:50.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:48:50.503 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T15:48:50.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T15:48:50.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:48:50.863 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:48:50.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T15:48:50.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:48:50.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:48:50.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:48:50.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:48:50.902 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:50.900+0000 7f66e5425640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:48:50.903 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:50.902+0000 7f66e5425640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:48:50.904 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:50.903+0000 7f66e5425640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:48:51.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T15:48:51.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:48:51.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:48:51.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:48:51.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:48:51.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:48:51.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:48:51.145 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:48:51.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:48:51.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:48:51.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:48:51.428 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:51.427+0000 7f66e5425640 -1 Falling back to public interface 2024-08-19T15:48:52.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:48:52.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:48:52.420 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:48:52.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:48:52.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:48:52.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:48:52.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:48:53.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:48:53.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:48:53.705 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:48:53.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:48:53.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:48:53.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:48:53.760 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:48:53.759+0000 7f66e5425640 -1 osd.0 0 log_to_monitors true 2024-08-19T15:48:54.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:48:55.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:48:55.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:48:55.029 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:48:55.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:48:55.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:48:55.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:48:55.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:48:56.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:48:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:48:56.319 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:48:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:48:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:48:56.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:48:56.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:48:57.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:48:57.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:48:57.607 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:48:57.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:48:57.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:48:57.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:48:57.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:48:58.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:48:58.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:48:58.895 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:48:58.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:48:58.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:48:58.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:48:59.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:00.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:00.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:00.182 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:49:00.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:49:00.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:00.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:49:00.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:01.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:01.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:01.500 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:49:01.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:49:01.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:01.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:49:01.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:02.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:02.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:02.789 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:49:02.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:49:02.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:02.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:49:03.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:04.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:04.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:04.079 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:49:04.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:49:04.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:04.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:49:04.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:05.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:05.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:05.368 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:49:05.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:49:05.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:05.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:49:06.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:07.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:07.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:07.036 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:49:07.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:49:07.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:07.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:49:07.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:08.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:08.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:08.647 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:49:08.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:49:08.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:08.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:49:09.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:10.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:10.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:10.257 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:49:10.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:49:10.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:10.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:49:10.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:11.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:11.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:11.749 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:49:11.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:49:11.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:11.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:49:12.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:13.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:13.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:13.471 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:49:13.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:49:13.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:13.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:49:14.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:15.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:15.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:15.141 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:49:15.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:49:15.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:15.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:49:15.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:16.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:16.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:16.872 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:49:16.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:49:16.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:16.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:49:17.158 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/961914007,v1:127.0.0.1:6803/961914007] [v2:127.0.0.1:6804/961914007,v1:127.0.0.1:6805/961914007] exists,up d3244e31-0aac-4fc0-aed6-bf16f2fc504b 2024-08-19T15:49:17.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:49:17.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:49:17.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:49:17.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:49:17.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2024-08-19T15:49:17.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-08-19T15:49:17.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:49:17.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T15:49:17.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:49:17.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/1 2024-08-19T15:49:17.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-08-19T15:49:17.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:49:17.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:49:17.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:49:17.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-08-19T15:49:17.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-08-19T15:49:17.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:49:17.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:49:17.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-08-19T15:49:17.164 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:49:17.164 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:49:17.164 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:49:17.164 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:49:17.164 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:49:17.164 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:49:17.164 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:49:17.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:49:17.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:49:17.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:49:17.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:49:17.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-08-19T15:49:17.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-08-19T15:49:17.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:49:17.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:49:17.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:49:17.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:49:17.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:49:17.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T15:49:17.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/1 2024-08-19T15:49:17.165 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:49:17.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8b31f940-cae1-402a-b136-e49528827c75 2024-08-19T15:49:17.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8b31f940-cae1-402a-b136-e49528827c75' 2024-08-19T15:49:17.166 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 8b31f940-cae1-402a-b136-e49528827c75 2024-08-19T15:49:17.166 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:49:17.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB9acNmQKUeCxAAQcOLaw8rRM8wnkdLakez3g== 2024-08-19T15:49:17.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB9acNmQKUeCxAAQcOLaw8rRM8wnkdLakez3g=="}' 2024-08-19T15:49:17.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8b31f940-cae1-402a-b136-e49528827c75 -i td/osd-scrub-snaps/1/new.json 2024-08-19T15:49:17.543 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:49:17.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/1/new.json 2024-08-19T15:49:17.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9acNmQKUeCxAAQcOLaw8rRM8wnkdLakez3g== --osd-uuid 8b31f940-cae1-402a-b136-e49528827c75 2024-08-19T15:49:17.594 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:49:17.592+0000 7fab5521e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:49:17.619 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:49:17.617+0000 7fab5521e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:49:17.620 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:49:17.619+0000 7fab5521e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:49:17.620 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:49:17.619+0000 7fab5521e640 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-08-19T15:49:17.620 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:49:17.619+0000 7fab5521e640 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-08-19T15:49:17.620 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:49:17.619+0000 7fab5521e640 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-08-19T15:49:17.694 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:49:17.693+0000 7fab5521e640 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2024-08-19T15:49:23.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2024-08-19T15:49:23.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:49:23.476 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T15:49:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T15:49:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:49:23.831 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:49:23.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T15:49:23.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:49:23.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:49:23.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:49:23.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:49:23.869 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:49:23.867+0000 7f0ca6e25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:49:23.870 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:49:23.869+0000 7f0ca6e25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:49:23.872 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:49:23.870+0000 7f0ca6e25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:49:24.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T15:49:24.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:49:24.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:49:24.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:49:24.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:49:24.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:24.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:49:24.155 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:49:24.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:24.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:49:24.410 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:49:24.409+0000 7f0ca6e25640 -1 Falling back to public interface 2024-08-19T15:49:24.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:25.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:25.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:25.433 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:49:25.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:49:25.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:25.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:49:25.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:26.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:26.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:26.715 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:49:26.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:49:26.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:26.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:49:26.752 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:49:26.750+0000 7f0ca6e25640 -1 osd.1 0 log_to_monitors true 2024-08-19T15:49:27.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:28.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:28.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:28.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:49:28.013 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:49:28.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:28.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:49:28.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:29.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:29.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:29.298 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T15:49:29.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T15:49:29.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:29.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:49:29.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:30.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:30.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:30.607 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T15:49:30.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T15:49:30.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:30.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:49:30.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:31.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:31.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:31.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T15:49:31.936 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T15:49:31.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:31.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:49:32.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:33.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:33.226 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T15:49:33.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:33.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T15:49:33.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:33.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:49:33.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:34.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:34.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:34.649 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T15:49:34.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T15:49:34.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:34.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:49:34.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:35.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:35.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:35.949 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T15:49:35.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T15:49:35.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:35.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:49:36.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:37.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:37.242 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T15:49:37.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:37.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T15:49:37.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:37.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:49:37.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:38.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:38.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:38.578 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T15:49:38.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T15:49:38.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:38.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:49:38.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:39.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:39.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:39.905 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T15:49:39.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T15:49:39.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:39.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:49:40.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:41.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:41.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:41.509 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T15:49:41.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T15:49:41.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:41.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:49:42.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:43.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:43.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:43.069 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T15:49:43.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T15:49:43.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:43.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:49:43.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:44.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:44.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:44.904 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T15:49:44.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T15:49:44.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:44.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:49:45.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:46.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:46.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:46.641 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T15:49:46.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T15:49:46.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:46.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:49:46.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:47.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:47.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:47.931 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T15:49:47.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T15:49:47.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:47.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:49:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:49:49.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:49:49.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:49:49.478 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T15:49:49.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T15:49:49.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:49:49.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:49:49.795 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1354029070,v1:127.0.0.1:6811/1354029070] [v2:127.0.0.1:6812/1354029070,v1:127.0.0.1:6813/1354029070] exists,up 8b31f940-cae1-402a-b136-e49528827c75 2024-08-19T15:49:49.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:49:49.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:49:49.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:49:49.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:748: _scrub_snaps_multi: ceph osd set noscrub 2024-08-19T15:49:50.171 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:49:50.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:749: _scrub_snaps_multi: ceph osd set nodeep-scrub 2024-08-19T15:49:50.648 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:49:50.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: create_pool test 1 1 2024-08-19T15:49:50.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-08-19T15:49:51.092 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' created 2024-08-19T15:49:51.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:49:52.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:753: _scrub_snaps_multi: wait_for_clean 2024-08-19T15:49:52.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:49:52.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:49:52.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:49:52.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:49:52.106 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:49:52.106 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:49:52.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:49:52.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:49:52.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:49:52.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:49:52.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:49:52.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:49:52.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:49:52.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:49:52.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:49:52.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:49:52.474 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:49:52.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:49:52.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:49:52.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:49:52.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-08-19T15:49:52.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-08-19T15:49:52.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-08-19T15:49:52.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:49:52.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:49:52.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-08-19T15:49:52.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-08-19T15:49:52.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672967' 2024-08-19T15:49:52.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:49:52.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-08-19T15:49:52.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:49:52.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:49:52.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-08-19T15:49:52.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:49:52.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-08-19T15:49:52.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-08-19T15:49:52.738 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836497 2024-08-19T15:49:52.738 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:49:53.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-08-19T15:49:53.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:49:53.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-08-19T15:49:53.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:49:53.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:49:53.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-08-19T15:49:53.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:49:53.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-08-19T15:49:53.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-08-19T15:49:53.083 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672967 2024-08-19T15:49:53.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:49:53.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672967 2024-08-19T15:49:53.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:49:54.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:49:54.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:49:54.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672967 2024-08-19T15:49:54.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:49:55.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:49:55.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:49:55.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-08-19T15:49:55.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:49:55.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:49:55.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:49:56.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:49:56.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:49:56.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:49:56.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:49:56.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:49:56.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:49:56.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:49:56.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:49:56.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:49:56.606 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:49:56.606 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:49:56.606 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:49:56.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:49:56.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:49:56.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:49:56.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: ceph osd dump 2024-08-19T15:49:56.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2024-08-19T15:49:56.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: awk '{ print $2 }' 2024-08-19T15:49:57.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: poolid=1 2024-08-19T15:49:57.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:756: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.470418 bs=1032 count=1 2024-08-19T15:49:57.238 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records in 2024-08-19T15:49:57.238 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records out 2024-08-19T15:49:57.238 INFO:tasks.workunit.client.0.smithi102.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.7477e-05 s, 21.7 MB/s 2024-08-19T15:49:57.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: seq 1 16 2024-08-19T15:49:57.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:49:57.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj1 testdata.470418 2024-08-19T15:49:57.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:49:57.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj2 testdata.470418 2024-08-19T15:49:57.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:49:57.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj3 testdata.470418 2024-08-19T15:49:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:49:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj4 testdata.470418 2024-08-19T15:49:57.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:49:57.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj5 testdata.470418 2024-08-19T15:49:57.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:49:57.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj6 testdata.470418 2024-08-19T15:49:58.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:49:58.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj7 testdata.470418 2024-08-19T15:49:58.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:49:58.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj8 testdata.470418 2024-08-19T15:49:58.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:49:58.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj9 testdata.470418 2024-08-19T15:49:58.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:49:58.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj10 testdata.470418 2024-08-19T15:49:58.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:49:58.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj11 testdata.470418 2024-08-19T15:49:58.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:49:58.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj12 testdata.470418 2024-08-19T15:49:58.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:49:58.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj13 testdata.470418 2024-08-19T15:49:59.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:49:59.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj14 testdata.470418 2024-08-19T15:49:59.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:49:59.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj15 testdata.470418 2024-08-19T15:49:59.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-08-19T15:49:59.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj16 testdata.470418 2024-08-19T15:49:59.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: get_primary test obj1 2024-08-19T15:49:59.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-08-19T15:49:59.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-08-19T15:49:59.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-08-19T15:49:59.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:49:59.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: local primary=1 2024-08-19T15:49:59.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: get_not_primary test obj1 2024-08-19T15:49:59.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-08-19T15:49:59.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-08-19T15:49:59.786 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-08-19T15:49:59.786 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-08-19T15:49:59.786 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-08-19T15:49:59.786 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-08-19T15:49:59.786 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:50:00.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T15:50:00.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-08-19T15:50:00.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T15:50:00.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: local replica=0 2024-08-19T15:50:00.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.470418 '$replica' 2024-08-19T15:50:00.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.470418 0 2024-08-19T15:50:00.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-08-19T15:50:00.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-08-19T15:50:00.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.470418 2024-08-19T15:50:00.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2024-08-19T15:50:00.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-08-19T15:50:00.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-08-19T15:50:00.578 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap1 2024-08-19T15:50:00.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=1 2024-08-19T15:50:00.581 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records in 2024-08-19T15:50:00.581 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records out 2024-08-19T15:50:00.581 INFO:tasks.workunit.client.0.smithi102.stderr:256 bytes copied, 6.191e-05 s, 4.1 MB/s 2024-08-19T15:50:00.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.470418 2024-08-19T15:50:00.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.470418 2024-08-19T15:50:00.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.470418 2024-08-19T15:50:01.133 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-08-19T15:50:01.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:50:01.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.470418 2024-08-19T15:50:01.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:50:01.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.470418 2024-08-19T15:50:01.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:50:01.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.470418 2024-08-19T15:50:01.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:50:01.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.470418 2024-08-19T15:50:01.693 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:50:01.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.470418 2024-08-19T15:50:01.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:50:01.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.470418 2024-08-19T15:50:02.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:50:02.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.470418 2024-08-19T15:50:02.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:50:02.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.470418 2024-08-19T15:50:02.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-08-19T15:50:02.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.470418 2024-08-19T15:50:02.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-08-19T15:50:02.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-08-19T15:50:02.580 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap2 2024-08-19T15:50:02.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=2 2024-08-19T15:50:02.583 INFO:tasks.workunit.client.0.smithi102.stderr:2+0 records in 2024-08-19T15:50:02.583 INFO:tasks.workunit.client.0.smithi102.stderr:2+0 records out 2024-08-19T15:50:02.583 INFO:tasks.workunit.client.0.smithi102.stderr:512 bytes copied, 6.5681e-05 s, 7.8 MB/s 2024-08-19T15:50:02.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.470418 2024-08-19T15:50:02.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-08-19T15:50:02.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-08-19T15:50:03.016 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap3 2024-08-19T15:50:03.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=3 2024-08-19T15:50:03.019 INFO:tasks.workunit.client.0.smithi102.stderr:3+0 records in 2024-08-19T15:50:03.019 INFO:tasks.workunit.client.0.smithi102.stderr:3+0 records out 2024-08-19T15:50:03.019 INFO:tasks.workunit.client.0.smithi102.stderr:768 bytes copied, 6.4864e-05 s, 11.8 MB/s 2024-08-19T15:50:03.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.470418 2024-08-19T15:50:03.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-08-19T15:50:03.269 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-08-19T15:50:03.427 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap4 2024-08-19T15:50:03.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=4 2024-08-19T15:50:03.429 INFO:tasks.workunit.client.0.smithi102.stderr:4+0 records in 2024-08-19T15:50:03.430 INFO:tasks.workunit.client.0.smithi102.stderr:4+0 records out 2024-08-19T15:50:03.430 INFO:tasks.workunit.client.0.smithi102.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 6.7608e-05 s, 15.1 MB/s 2024-08-19T15:50:03.430 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.470418 2024-08-19T15:50:03.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.470418 2024-08-19T15:50:03.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-08-19T15:50:03.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-08-19T15:50:03.980 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap5 2024-08-19T15:50:03.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-08-19T15:50:03.982 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-08-19T15:50:04.222 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap6 2024-08-19T15:50:04.224 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=6 2024-08-19T15:50:04.225 INFO:tasks.workunit.client.0.smithi102.stderr:6+0 records in 2024-08-19T15:50:04.225 INFO:tasks.workunit.client.0.smithi102.stderr:6+0 records out 2024-08-19T15:50:04.225 INFO:tasks.workunit.client.0.smithi102.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 7.0592e-05 s, 21.8 MB/s 2024-08-19T15:50:04.225 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.470418 2024-08-19T15:50:04.466 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-08-19T15:50:04.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-08-19T15:50:04.624 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap7 2024-08-19T15:50:04.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-08-19T15:50:04.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-08-19T15:50:05.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-08-19T15:50:05.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-08-19T15:50:05.229 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:50:05.230 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:50:05.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:50:05.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:50:05.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:50:05.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:50:05.536 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2024-08-19T15:50:06.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:06.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-08-19T15:50:07.611 INFO:tasks.workunit.client.0.smithi102.stdout:WARNING: only removing head with clones present 2024-08-19T15:50:07.611 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:ff7b1f36:::obj1:head# 2024-08-19T15:50:08.127 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-08-19T15:50:08.127 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-08-19T15:50:09.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:09.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-08-19T15:50:09.665 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:c52c9666:::obj5:2# 2024-08-19T15:50:10.184 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-08-19T15:50:10.184 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-08-19T15:50:11.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:11.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:11.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-08-19T15:50:12.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-08-19T15:50:12.127 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:50:12.127 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-08-19T15:50:12.127 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:12.127 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-08-19T15:50:12.127 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-08-19T15:50:12.127 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-08-19T15:50:12.127 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-08-19T15:50:12.127 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-08-19T15:50:12.128 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-08-19T15:50:12.128 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-08-19T15:50:12.128 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:50:12.128 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-08-19T15:50:12.128 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-08-19T15:50:12.128 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-08-19T15:50:12.128 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-08-19T15:50:12.128 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:50:12.128 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:50:12.128 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:12.129 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:50:12.129 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:50:12.129 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:50:12.129 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:12.129 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:50:12.129 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:50:12.129 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:50:12.129 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:12.129 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:50:12.129 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:50:12.129 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:50:12.129 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:12.129 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:50:12.130 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:50:12.130 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:50:12.130 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:12.130 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:50:12.130 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:50:12.130 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:50:12.130 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:12.130 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:50:12.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-08-19T15:50:12.131 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-08-19T15:50:12.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-08-19T15:50:12.670 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:c52c9666:::obj5:1# 2024-08-19T15:50:13.184 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-08-19T15:50:14.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-08-19T15:50:14.232 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:50:14.232 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-08-19T15:50:14.232 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:14.232 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-08-19T15:50:14.232 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-08-19T15:50:14.232 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-08-19T15:50:14.232 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-08-19T15:50:14.233 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-08-19T15:50:14.233 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-08-19T15:50:14.233 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-08-19T15:50:14.233 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:50:14.233 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-08-19T15:50:14.233 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-08-19T15:50:14.233 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-08-19T15:50:14.233 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-08-19T15:50:14.233 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:50:14.233 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:50:14.233 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:14.233 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:50:14.233 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:50:14.234 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:50:14.234 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:14.234 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:50:14.234 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:50:14.234 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:50:14.234 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:14.234 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:50:14.234 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:50:14.234 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:50:14.234 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:14.234 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:50:14.234 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:50:14.234 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:50:14.235 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:14.235 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:50:14.235 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:50:14.235 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-08-19T15:50:14.235 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:14.235 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:50:14.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-08-19T15:50:14.235 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-08-19T15:50:14.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-08-19T15:50:14.236 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-08-19T15:50:14.236 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-08-19T15:50:14.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:14.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=18 2024-08-19T15:50:14.989 INFO:tasks.workunit.client.0.smithi102.stderr:18+0 records in 2024-08-19T15:50:14.989 INFO:tasks.workunit.client.0.smithi102.stderr:18+0 records out 2024-08-19T15:50:14.989 INFO:tasks.workunit.client.0.smithi102.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000109889 s, 41.9 MB/s 2024-08-19T15:50:14.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.470418 2024-08-19T15:50:16.082 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2024-08-19T15:50:17.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:17.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=15 2024-08-19T15:50:17.697 INFO:tasks.workunit.client.0.smithi102.stderr:15+0 records in 2024-08-19T15:50:17.697 INFO:tasks.workunit.client.0.smithi102.stderr:15+0 records out 2024-08-19T15:50:17.697 INFO:tasks.workunit.client.0.smithi102.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000100423 s, 38.2 MB/s 2024-08-19T15:50:17.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.470418 2024-08-19T15:50:18.789 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2024-08-19T15:50:18.789 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-08-19T15:50:19.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:19.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-08-19T15:50:20.260 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:0ee9ae15:::obj4:7# 2024-08-19T15:50:20.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-08-19T15:50:21.834 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-08-19T15:50:21.835 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-08-19T15:50:21.835 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:21.835 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-08-19T15:50:21.835 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-08-19T15:50:21.835 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-08-19T15:50:21.835 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-08-19T15:50:21.835 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-08-19T15:50:21.835 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-08-19T15:50:21.835 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-08-19T15:50:21.835 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:50:21.835 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-08-19T15:50:21.836 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-08-19T15:50:21.836 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-08-19T15:50:21.836 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-08-19T15:50:21.836 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:50:21.836 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:21.836 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:50:21.836 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:50:21.836 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:21.836 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:50:21.836 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:50:21.836 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:21.836 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:50:21.836 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:50:21.836 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:21.837 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:50:21.837 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:50:21.837 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:21.837 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:50:21.837 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:50:21.837 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:21.837 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:50:21.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-08-19T15:50:21.837 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:21.838 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:21.838 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:21.838 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:21.838 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:21.838 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:21.838 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-08-19T15:50:21.838 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2024-08-19T15:50:21.838 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-08-19T15:50:22.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:22.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-08-19T15:50:23.165 INFO:tasks.workunit.client.0.smithi102.stdout:remove #1:461f8b5e:::obj16:7# 2024-08-19T15:50:23.680 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-08-19T15:50:24.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-08-19T15:50:24.731 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-08-19T15:50:24.731 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-08-19T15:50:24.731 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-08-19T15:50:24.731 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-08-19T15:50:24.731 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-08-19T15:50:24.731 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-08-19T15:50:24.731 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-08-19T15:50:24.731 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-08-19T15:50:24.732 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:50:24.732 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-08-19T15:50:24.732 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-08-19T15:50:24.732 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-08-19T15:50:24.732 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-08-19T15:50:24.732 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:50:24.732 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:50:24.732 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-08-19T15:50:24.732 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:50:24.732 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:50:24.732 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-08-19T15:50:24.732 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-08-19T15:50:24.732 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:50:24.732 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:50:24.733 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-08-19T15:50:24.733 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:50:24.733 INFO:tasks.workunit.client.0.smithi102.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-08-19T15:50:24.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-08-19T15:50:24.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-08-19T15:50:24.750 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2024-08-19T15:50:25.536 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:25.536 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-08-19T15:50:26.602 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:26.602 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-08-19T15:50:26.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:26.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.470418 bs=256 count=7 2024-08-19T15:50:26.604 INFO:tasks.workunit.client.0.smithi102.stderr:7+0 records in 2024-08-19T15:50:26.604 INFO:tasks.workunit.client.0.smithi102.stderr:7+0 records out 2024-08-19T15:50:26.604 INFO:tasks.workunit.client.0.smithi102.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 0.000114959 s, 15.6 MB/s 2024-08-19T15:50:26.604 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.470418 2024-08-19T15:50:27.886 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2024-08-19T15:50:28.599 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:50:28.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:28.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-08-19T15:50:29.932 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2024-08-19T15:50:30.621 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:50:31.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:31.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-08-19T15:50:32.449 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2024-08-19T15:50:33.209 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:50:33.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:33.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-08-19T15:50:34.492 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2024-08-19T15:50:35.237 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:50:36.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:36.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-08-19T15:50:37.074 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2024-08-19T15:50:37.733 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:50:38.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:38.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-08-19T15:50:39.032 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2024-08-19T15:50:39.703 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:50:39.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:39.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-08-19T15:50:41.019 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2024-08-19T15:50:41.690 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:50:41.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:41.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-08-19T15:50:43.006 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2024-08-19T15:50:43.726 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:50:43.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:43.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-08-19T15:50:45.043 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2024-08-19T15:50:45.701 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:50:46.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:46.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-08-19T15:50:47.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-08-19T15:50:47.336 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2024-08-19T15:50:48.002 INFO:tasks.workunit.client.0.smithi102.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-08-19T15:50:48.265 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-08-19T15:50:48.265 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-08-19T15:50:49.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-08-19T15:50:49.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-08-19T15:50:49.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: rm -f testdata.470418 2024-08-19T15:50:49.408 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: expr 2 - 1 2024-08-19T15:50:49.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: seq 0 1 2024-08-19T15:50:49.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:50:49.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2024-08-19T15:50:49.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-08-19T15:50:49.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:50:49.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T15:50:49.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:50:49.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-08-19T15:50:49.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-08-19T15:50:49.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:50:49.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:50:49.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:50:49.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-08-19T15:50:49.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-08-19T15:50:49.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:50:49.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:50:49.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-08-19T15:50:49.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:50:49.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:50:49.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:50:49.414 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:50:49.414 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:50:49.414 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:50:49.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:50:49.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:50:49.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:50:49.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-08-19T15:50:49.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-08-19T15:50:49.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:50:49.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:50:49.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:50:49.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:50:49.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:50:49.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:50:49.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-08-19T15:50:49.416 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:50:49.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T15:50:49.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:50:49.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-08-19T15:50:49.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T15:50:49.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:50:49.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:50:49.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:50:49.454 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:50:49.453+0000 7f5cc0c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:50:49.456 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:50:49.455+0000 7f5cc0c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:50:49.457 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:50:49.456+0000 7f5cc0c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:50:49.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T15:50:49.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:50:49.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:50:49.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:50:49.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:50:49.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:50:49.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:50:49.700 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:50:49.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:50:49.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:50:49.974 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:50:49.973+0000 7f5cc0c0f640 -1 Falling back to public interface 2024-08-19T15:50:50.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:50:51.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:50:51.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:50:51.021 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:50:51.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:50:51.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:50:51.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:50:51.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:50:52.199 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:50:52.197+0000 7f5cc0c0f640 -1 osd.0 23 log_to_monitors true 2024-08-19T15:50:52.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:50:52.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:50:52.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:50:52.319 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:50:52.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:50:52.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:50:52.649 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 26 up_thru 0 down_at 24 last_clean_interval [5,23) [v2:127.0.0.1:6802/2595406120,v1:127.0.0.1:6803/2595406120] [v2:127.0.0.1:6804/2595406120,v1:127.0.0.1:6805/2595406120] exists,up d3244e31-0aac-4fc0-aed6-bf16f2fc504b 2024-08-19T15:50:52.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:50:52.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:50:52.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:50:52.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:50:52.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2024-08-19T15:50:52.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-08-19T15:50:52.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T15:50:52.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-19T15:50:52.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T15:50:52.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/1 2024-08-19T15:50:52.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-08-19T15:50:52.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:50:52.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:50:52.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:50:52.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-08-19T15:50:52.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-08-19T15:50:52.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T15:50:52.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T15:50:52.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-08-19T15:50:52.655 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T15:50:52.655 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:50:52.655 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:50:52.655 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:50:52.655 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:50:52.655 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:50:52.655 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:50:52.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' 2024-08-19T15:50:52.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:50:52.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-08-19T15:50:52.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-08-19T15:50:52.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:50:52.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:50:52.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:50:52.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:50:52.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T15:50:52.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T15:50:52.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/1 2024-08-19T15:50:52.656 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:50:52.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-19T15:50:52.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.470418/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T15:50:52.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/1/whoami 2024-08-19T15:50:52.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-19T15:50:52.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T15:50:52.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T15:50:52.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T15:50:52.690 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:50:52.688+0000 7f5357a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:50:52.692 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:50:52.691+0000 7f5357a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:50:52.693 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:50:52.692+0000 7f5357a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:50:52.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-19T15:50:52.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:50:52.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:50:52.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:50:52.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:50:52.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:50:52.958 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:50:52.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:50:52.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:50:52.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:50:53.208 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:50:53.207+0000 7f5357a11640 -1 Falling back to public interface 2024-08-19T15:50:53.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:50:54.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:50:54.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:50:54.272 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:50:54.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:50:54.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:50:54.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:50:54.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:50:55.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:50:55.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:50:55.563 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:50:55.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:50:55.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:50:55.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:50:55.835 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:50:55.833+0000 7f5357a11640 -1 osd.1 23 log_to_monitors true 2024-08-19T15:50:55.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:50:56.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:50:56.848 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:50:56.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:50:56.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:50:56.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:50:56.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:50:57.126 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6810/24690184,v1:127.0.0.1:6811/24690184] [v2:127.0.0.1:6812/24690184,v1:127.0.0.1:6813/24690184] exists,up 8b31f940-cae1-402a-b136-e49528827c75 2024-08-19T15:50:57.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:50:57.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:50:57.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:50:57.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2024-08-19T15:50:57.275 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: { 2024-08-19T15:50:57.275 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-08-19T15:50:57.275 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:50:57.288 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: { 2024-08-19T15:50:57.288 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-08-19T15:50:57.288 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:50:57.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:775: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 3 2024-08-19T15:50:57.448 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: { 2024-08-19T15:50:57.448 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-08-19T15:50:57.448 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:50:57.461 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: { 2024-08-19T15:50:57.461 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-08-19T15:50:57.461 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:50:57.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_scrub_chunk_min 3 2024-08-19T15:50:57.578 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: { 2024-08-19T15:50:57.578 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-08-19T15:50:57.578 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:50:57.592 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: { 2024-08-19T15:50:57.592 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-08-19T15:50:57.592 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:50:57.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:777: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2024-08-19T15:50:57.742 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: { 2024-08-19T15:50:57.742 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-08-19T15:50:57.742 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:50:57.756 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: { 2024-08-19T15:50:57.756 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-08-19T15:50:57.756 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:50:57.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2024-08-19T15:50:57.893 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: { 2024-08-19T15:50:57.893 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-08-19T15:50:57.893 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:50:57.906 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: { 2024-08-19T15:50:57.906 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-08-19T15:50:57.906 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:50:57.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:779: _scrub_snaps_multi: wait_for_clean 2024-08-19T15:50:57.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:50:57.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:50:57.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:50:57.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:50:57.916 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:50:57.917 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:50:57.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:50:57.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:50:57.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:50:58.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:50:58.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:50:58.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:50:58.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:50:58.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:50:58.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:50:58.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:50:58.282 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T15:50:58.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:50:58.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:50:58.282 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:50:58.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149699 2024-08-19T15:50:58.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149699 2024-08-19T15:50:58.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149699' 2024-08-19T15:50:58.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:50:58.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:50:58.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-08-19T15:50:58.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-08-19T15:50:58.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149699 1-124554051586' 2024-08-19T15:50:58.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:50:58.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149699 2024-08-19T15:50:58.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:50:58.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:50:58.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149699 2024-08-19T15:50:58.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:50:58.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149699 2024-08-19T15:50:58.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149699' 2024-08-19T15:50:58.560 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 111669149699 2024-08-19T15:50:58.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:50:58.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149699 -lt 111669149699 2024-08-19T15:50:58.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:50:58.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-124554051586 2024-08-19T15:50:58.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:50:58.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:50:58.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-124554051586 2024-08-19T15:50:58.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:50:58.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-08-19T15:50:58.867 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 124554051586 2024-08-19T15:50:58.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-08-19T15:50:58.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:50:59.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-08-19T15:50:59.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:50:59.146 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:50:59.146 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:50:59.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:50:59.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:50:59.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:50:59.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:50:59.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:50:59.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:50:59.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:50:59.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:50:59.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:50:59.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:50:59.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:50:59.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:51:00.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:51:00.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:51:00.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:51:00.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:781: _scrub_snaps_multi: local pgid=1.0 2024-08-19T15:51:00.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:782: _scrub_snaps_multi: pg_scrub 1.0 2024-08-19T15:51:00.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=1.0 2024-08-19T15:51:00.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 1.0 2024-08-19T15:51:00.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:51:00.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:51:00.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:51:00.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:51:00.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-08-19T15:49:50.948603+0000 2024-08-19T15:51:00.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 1.0 2024-08-19T15:51:00.720 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 1.0 on osd.1 to scrub 2024-08-19T15:51:00.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 1.0 2024-08-19T15:49:50.948603+0000 2024-08-19T15:51:00.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-08-19T15:51:00.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-19T15:49:50.948603+0000 2024-08-19T15:51:00.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:51:00.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:51:00.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:51:00.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:51:00.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:51:00.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:51:00.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:51:00.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:51:01.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:49:50.948603+0000 '>' 2024-08-19T15:49:50.948603+0000 2024-08-19T15:51:01.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:51:02.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:51:02.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:51:02.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:51:02.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:51:02.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:51:02.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:51:02.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:51:02.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:49:50.948603+0000 '>' 2024-08-19T15:49:50.948603+0000 2024-08-19T15:51:02.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:51:03.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:51:03.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:51:03.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:51:03.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:51:03.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:51:03.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:51:03.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:51:03.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:49:50.948603+0000 '>' 2024-08-19T15:49:50.948603+0000 2024-08-19T15:51:03.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:51:04.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:51:04.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:51:04.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:51:04.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:51:04.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:51:04.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:51:04.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:51:04.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:49:50.948603+0000 '>' 2024-08-19T15:49:50.948603+0000 2024-08-19T15:51:04.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:51:05.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:51:05.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:51:05.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:51:05.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:51:05.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:51:05.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:51:05.880 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:51:06.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:49:50.948603+0000 '>' 2024-08-19T15:49:50.948603+0000 2024-08-19T15:51:06.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:51:07.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:51:07.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-08-19T15:51:07.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:51:07.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:51:07.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:51:07.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:51:07.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:51:07.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T15:51:01.469199+0000 '>' 2024-08-19T15:49:50.948603+0000 2024-08-19T15:51:07.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T15:51:07.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2024-08-19T15:51:07.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: wc -l 2024-08-19T15:51:07.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: test 16 -gt 3 2024-08-19T15:51:07.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:51:07.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: wc -l 2024-08-19T15:51:07.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: test 16 -gt 3 2024-08-19T15:51:07.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:789: _scrub_snaps_multi: rados list-inconsistent-pg test 2024-08-19T15:51:07.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2024-08-19T15:51:07.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: test 1 = 1 2024-08-19T15:51:07.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2024-08-19T15:51:07.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: test 1.0 = 1.0 2024-08-19T15:51:07.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:795: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2024-08-19T15:51:07.536 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:51:07.536 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 29, 2024-08-19T15:51:07.537 INFO:tasks.workunit.client.0.smithi102.stdout: "inconsistents": [ 2024-08-19T15:51:07.537 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.537 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:51:07.537 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj5", 2024-08-19T15:51:07.537 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:51:07.537 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:51:07.537 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:51:07.537 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 5 2024-08-19T15:51:07.537 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.537 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.537 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [ 2024-08-19T15:51:07.537 INFO:tasks.workunit.client.0.smithi102.stdout: "missing" 2024-08-19T15:51:07.538 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.538 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:51:07.538 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:51:07.538 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj5", 2024-08-19T15:51:07.538 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:51:07.538 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": 1, 2024-08-19T15:51:07.538 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 1718170787, 2024-08-19T15:51:07.538 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:51:07.538 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:51:07.538 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:51:07.538 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.538 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'19", 2024-08-19T15:51:07.538 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'5", 2024-08-19T15:51:07.539 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4243.0:1", 2024-08-19T15:51:07.539 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 5, 2024-08-19T15:51:07.539 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:51:07.539 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:49:57.836482+0000", 2024-08-19T15:51:07.539 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:49:57.837022+0000", 2024-08-19T15:51:07.539 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:51:07.539 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:51:07.539 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:51:07.539 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:51:07.539 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.539 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:51:07.539 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:51:07.539 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0x87f5ff96", 2024-08-19T15:51:07.540 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:51:07.540 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:51:07.540 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:51:07.540 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:51:07.540 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:51:07.540 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:51:07.540 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.540 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:51:07.540 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.540 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:51:07.540 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.540 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:51:07.540 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:51:07.541 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.541 INFO:tasks.workunit.client.0.smithi102.stdout: "missing" 2024-08-19T15:51:07.541 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.541 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.541 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.541 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:51:07.541 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:51:07.541 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.541 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032 2024-08-19T15:51:07.541 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.541 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.541 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.541 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.541 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:51:07.542 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj5", 2024-08-19T15:51:07.542 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:51:07.542 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:51:07.542 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 2, 2024-08-19T15:51:07.542 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 20 2024-08-19T15:51:07.542 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.542 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.542 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [ 2024-08-19T15:51:07.542 INFO:tasks.workunit.client.0.smithi102.stdout: "missing" 2024-08-19T15:51:07.542 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.542 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:51:07.542 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:51:07.543 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj5", 2024-08-19T15:51:07.543 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:51:07.543 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": 2, 2024-08-19T15:51:07.543 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 1718170787, 2024-08-19T15:51:07.543 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:51:07.543 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:51:07.543 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:51:07.543 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.543 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "18'41", 2024-08-19T15:51:07.543 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "17'20", 2024-08-19T15:51:07.543 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4291.0:1", 2024-08-19T15:51:07.543 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 20, 2024-08-19T15:51:07.543 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.543 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:50:00.858059+0000", 2024-08-19T15:51:07.544 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:50:00.858622+0000", 2024-08-19T15:51:07.544 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:51:07.544 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:51:07.544 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:51:07.544 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:51:07.544 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.544 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:51:07.544 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:51:07.544 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0xc796de79", 2024-08-19T15:51:07.544 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:51:07.544 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:51:07.544 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:51:07.545 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:51:07.545 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:51:07.545 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:51:07.545 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.545 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:51:07.545 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.545 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:51:07.545 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.545 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:51:07.545 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:51:07.545 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.545 INFO:tasks.workunit.client.0.smithi102.stdout: "missing" 2024-08-19T15:51:07.545 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.545 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.546 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.546 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:51:07.546 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:51:07.546 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.546 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256 2024-08-19T15:51:07.546 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.546 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.546 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.546 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.546 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:51:07.546 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj5", 2024-08-19T15:51:07.546 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:51:07.547 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:51:07.547 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 4, 2024-08-19T15:51:07.547 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 42 2024-08-19T15:51:07.547 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.547 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.547 INFO:tasks.workunit.client.0.smithi102.stdout: "size_mismatch" 2024-08-19T15:51:07.547 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.547 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [ 2024-08-19T15:51:07.547 INFO:tasks.workunit.client.0.smithi102.stdout: "size_mismatch_info", 2024-08-19T15:51:07.547 INFO:tasks.workunit.client.0.smithi102.stdout: "obj_size_info_mismatch" 2024-08-19T15:51:07.547 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.547 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:51:07.547 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:51:07.547 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj5", 2024-08-19T15:51:07.548 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:51:07.548 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": 4, 2024-08-19T15:51:07.548 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 1718170787, 2024-08-19T15:51:07.548 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:51:07.548 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:51:07.548 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:51:07.548 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.548 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "20'45", 2024-08-19T15:51:07.548 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "18'42", 2024-08-19T15:51:07.548 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4327.0:1", 2024-08-19T15:51:07.548 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 42, 2024-08-19T15:51:07.548 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 512, 2024-08-19T15:51:07.548 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:50:02.618669+0000", 2024-08-19T15:51:07.549 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:50:02.637115+0000", 2024-08-19T15:51:07.549 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:51:07.549 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:51:07.549 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:51:07.549 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:51:07.549 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.549 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:51:07.550 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:51:07.550 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0xbd203d35", 2024-08-19T15:51:07.550 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:51:07.550 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:51:07.550 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:51:07.550 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:51:07.550 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:51:07.550 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:51:07.550 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.550 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:51:07.550 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.550 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:51:07.551 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.551 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:51:07.551 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:51:07.551 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.551 INFO:tasks.workunit.client.0.smithi102.stdout: "size_mismatch_info", 2024-08-19T15:51:07.551 INFO:tasks.workunit.client.0.smithi102.stdout: "obj_size_info_mismatch" 2024-08-19T15:51:07.551 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.551 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 4608, 2024-08-19T15:51:07.551 INFO:tasks.workunit.client.0.smithi102.stdout: "object_info": { 2024-08-19T15:51:07.551 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:51:07.551 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj5", 2024-08-19T15:51:07.551 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:51:07.551 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": 4, 2024-08-19T15:51:07.551 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 1718170787, 2024-08-19T15:51:07.552 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:51:07.552 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:51:07.552 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:51:07.552 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.552 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "20'45", 2024-08-19T15:51:07.552 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "18'42", 2024-08-19T15:51:07.552 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4327.0:1", 2024-08-19T15:51:07.552 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 42, 2024-08-19T15:51:07.552 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 512, 2024-08-19T15:51:07.552 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:50:02.618669+0000", 2024-08-19T15:51:07.552 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:50:02.637115+0000", 2024-08-19T15:51:07.552 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:51:07.553 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:51:07.553 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:51:07.553 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:51:07.553 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.553 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:51:07.553 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:51:07.553 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0xbd203d35", 2024-08-19T15:51:07.553 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:51:07.553 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:51:07.553 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:51:07.553 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:51:07.553 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:51:07.553 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:51:07.553 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.554 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:51:07.554 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.554 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.554 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.554 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:51:07.554 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:51:07.554 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.554 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 512 2024-08-19T15:51:07.554 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.554 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.554 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.554 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.554 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:51:07.555 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj4", 2024-08-19T15:51:07.555 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:51:07.555 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:51:07.555 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 7, 2024-08-19T15:51:07.555 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 4 2024-08-19T15:51:07.555 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.555 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.555 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [ 2024-08-19T15:51:07.555 INFO:tasks.workunit.client.0.smithi102.stdout: "missing" 2024-08-19T15:51:07.555 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.555 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:51:07.555 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:51:07.555 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj4", 2024-08-19T15:51:07.556 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:51:07.556 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": 7, 2024-08-19T15:51:07.556 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 2826278768, 2024-08-19T15:51:07.556 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:51:07.556 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:51:07.556 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:51:07.556 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.556 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "23'51", 2024-08-19T15:51:07.556 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'4", 2024-08-19T15:51:07.556 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4240.0:1", 2024-08-19T15:51:07.556 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 4, 2024-08-19T15:51:07.556 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:51:07.556 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:49:57.702157+0000", 2024-08-19T15:51:07.557 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:49:57.702717+0000", 2024-08-19T15:51:07.557 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:51:07.557 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:51:07.557 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:51:07.557 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:51:07.557 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.557 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:51:07.557 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:51:07.557 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0x87f5ff96", 2024-08-19T15:51:07.558 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:51:07.558 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:51:07.558 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:51:07.558 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:51:07.558 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:51:07.558 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:51:07.558 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.558 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:51:07.558 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.558 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:51:07.559 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.559 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:51:07.559 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:51:07.559 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.559 INFO:tasks.workunit.client.0.smithi102.stdout: "missing" 2024-08-19T15:51:07.559 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.559 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.559 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.559 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:51:07.559 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:51:07.559 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.559 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032 2024-08-19T15:51:07.559 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.560 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.560 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.560 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.560 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:51:07.560 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj5", 2024-08-19T15:51:07.560 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:51:07.560 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:51:07.560 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 7, 2024-08-19T15:51:07.560 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 0 2024-08-19T15:51:07.560 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.560 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.560 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [ 2024-08-19T15:51:07.560 INFO:tasks.workunit.client.0.smithi102.stdout: "missing", 2024-08-19T15:51:07.560 INFO:tasks.workunit.client.0.smithi102.stdout: "info_missing" 2024-08-19T15:51:07.561 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.561 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:51:07.561 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.561 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:51:07.561 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:51:07.561 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.561 INFO:tasks.workunit.client.0.smithi102.stdout: "info_missing" 2024-08-19T15:51:07.561 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.561 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1792 2024-08-19T15:51:07.561 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.561 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.561 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:51:07.562 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:51:07.562 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.562 INFO:tasks.workunit.client.0.smithi102.stdout: "missing" 2024-08-19T15:51:07.562 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.562 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.562 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.562 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.562 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.562 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:51:07.562 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj1", 2024-08-19T15:51:07.562 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:51:07.562 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:51:07.562 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:51:07.563 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 18 2024-08-19T15:51:07.563 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.563 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.563 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [ 2024-08-19T15:51:07.563 INFO:tasks.workunit.client.0.smithi102.stdout: "missing" 2024-08-19T15:51:07.563 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.563 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:51:07.563 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:51:07.563 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj1", 2024-08-19T15:51:07.563 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:51:07.563 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:51:07.563 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 1828249343, 2024-08-19T15:51:07.563 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:51:07.563 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:51:07.564 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:51:07.564 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.564 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'18", 2024-08-19T15:51:07.564 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'1", 2024-08-19T15:51:07.564 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4288.0:1", 2024-08-19T15:51:07.564 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 18, 2024-08-19T15:51:07.564 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.564 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:50:00.616383+0000", 2024-08-19T15:51:07.564 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:50:00.711436+0000", 2024-08-19T15:51:07.564 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:51:07.564 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:51:07.564 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:51:07.564 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:51:07.565 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.565 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:51:07.565 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:51:07.565 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0xc796de79", 2024-08-19T15:51:07.565 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:51:07.565 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:51:07.565 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:51:07.565 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:51:07.565 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:51:07.565 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:51:07.565 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.565 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:51:07.565 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.565 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:51:07.566 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.566 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:51:07.566 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:51:07.566 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.566 INFO:tasks.workunit.client.0.smithi102.stdout: "missing" 2024-08-19T15:51:07.566 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.566 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.566 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.566 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:51:07.566 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:51:07.566 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.566 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256 2024-08-19T15:51:07.566 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.567 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.567 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.567 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.567 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:51:07.567 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj10", 2024-08-19T15:51:07.567 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:51:07.567 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:51:07.567 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:51:07.567 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 32 2024-08-19T15:51:07.567 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.567 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.567 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_inconsistency" 2024-08-19T15:51:07.567 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.567 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [], 2024-08-19T15:51:07.568 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:51:07.568 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:51:07.568 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj10", 2024-08-19T15:51:07.568 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:51:07.568 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:51:07.568 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 718195851, 2024-08-19T15:51:07.568 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:51:07.568 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:51:07.568 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:51:07.568 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.568 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'32", 2024-08-19T15:51:07.568 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'10", 2024-08-19T15:51:07.568 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4309.0:1", 2024-08-19T15:51:07.569 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 32, 2024-08-19T15:51:07.569 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.569 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:50:01.728383+0000", 2024-08-19T15:51:07.569 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:50:01.728925+0000", 2024-08-19T15:51:07.569 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:51:07.569 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:51:07.569 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:51:07.569 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:51:07.569 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.569 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:51:07.569 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:51:07.570 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0xc796de79", 2024-08-19T15:51:07.570 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:51:07.570 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:51:07.570 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:51:07.570 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:51:07.570 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:51:07.570 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:51:07.570 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.570 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:51:07.570 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.570 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:51:07.570 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.570 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:51:07.571 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:51:07.571 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.571 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.571 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:51:07.571 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:51:07.571 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:51:07.571 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.571 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:51:07.571 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:51:07.571 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "????", 2024-08-19T15:51:07.571 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:51:07.571 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:51:07.571 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.571 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.572 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.572 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.572 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.572 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.572 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:51:07.572 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:51:07.572 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.572 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.572 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:51:07.572 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:51:07.572 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:51:07.572 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.572 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:51:07.573 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:51:07.573 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:51:07.573 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:51:07.573 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:51:07.573 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.573 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.573 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.573 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.573 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.573 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.573 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.573 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.574 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:51:07.574 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj11", 2024-08-19T15:51:07.574 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:51:07.574 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:51:07.574 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:51:07.574 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 34 2024-08-19T15:51:07.574 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.574 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.574 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_inconsistency" 2024-08-19T15:51:07.574 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.574 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [], 2024-08-19T15:51:07.574 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:51:07.574 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:51:07.575 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj11", 2024-08-19T15:51:07.575 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:51:07.575 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:51:07.575 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 693400951, 2024-08-19T15:51:07.575 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:51:07.575 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:51:07.575 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:51:07.575 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.575 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'34", 2024-08-19T15:51:07.575 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'11", 2024-08-19T15:51:07.575 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4312.0:1", 2024-08-19T15:51:07.575 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 34, 2024-08-19T15:51:07.575 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.575 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:50:01.908552+0000", 2024-08-19T15:51:07.576 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:50:01.909115+0000", 2024-08-19T15:51:07.576 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:51:07.576 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:51:07.576 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:51:07.576 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:51:07.576 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.576 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:51:07.576 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:51:07.576 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0xc796de79", 2024-08-19T15:51:07.576 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:51:07.576 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:51:07.576 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:51:07.576 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:51:07.576 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:51:07.577 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:51:07.577 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.577 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:51:07.577 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.577 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:51:07.577 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.577 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:51:07.577 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:51:07.577 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.577 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.577 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:51:07.577 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:51:07.577 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [] 2024-08-19T15:51:07.578 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.578 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.578 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.578 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:51:07.578 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:51:07.578 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.578 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.578 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:51:07.578 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:51:07.578 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:51:07.578 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.578 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:51:07.578 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:51:07.579 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:51:07.579 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:51:07.579 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:51:07.579 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.579 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.579 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.579 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.579 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.579 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.579 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.579 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.579 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:51:07.579 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj13", 2024-08-19T15:51:07.579 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:51:07.580 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:51:07.580 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:51:07.580 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 38 2024-08-19T15:51:07.580 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.580 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.580 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_inconsistency" 2024-08-19T15:51:07.580 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.580 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [], 2024-08-19T15:51:07.580 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:51:07.580 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:51:07.580 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj13", 2024-08-19T15:51:07.580 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:51:07.580 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:51:07.580 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 2087409765, 2024-08-19T15:51:07.581 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:51:07.581 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:51:07.581 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:51:07.581 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.581 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'38", 2024-08-19T15:51:07.581 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'13", 2024-08-19T15:51:07.581 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4318.0:1", 2024-08-19T15:51:07.581 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 38, 2024-08-19T15:51:07.581 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.581 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:50:02.171880+0000", 2024-08-19T15:51:07.581 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:50:02.172451+0000", 2024-08-19T15:51:07.582 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:51:07.582 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:51:07.582 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:51:07.582 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:51:07.582 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.582 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:51:07.583 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:51:07.583 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0xc796de79", 2024-08-19T15:51:07.583 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:51:07.583 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:51:07.583 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:51:07.583 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:51:07.584 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:51:07.584 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:51:07.584 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.584 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:51:07.584 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.584 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:51:07.584 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.584 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:51:07.584 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:51:07.584 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.584 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.584 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:51:07.584 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:51:07.585 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:51:07.585 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.585 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:51:07.585 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:51:07.585 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]" 2024-08-19T15:51:07.585 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.585 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.585 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.585 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.585 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.585 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:51:07.585 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:51:07.585 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.586 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.586 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:51:07.586 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:51:07.586 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:51:07.586 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.586 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:51:07.586 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:51:07.586 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:51:07.586 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:51:07.586 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:51:07.587 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.587 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.587 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.587 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.587 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.587 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.587 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.587 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.587 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:51:07.587 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj14", 2024-08-19T15:51:07.587 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:51:07.587 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:51:07.587 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:51:07.588 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 40 2024-08-19T15:51:07.588 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.588 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.588 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_inconsistency" 2024-08-19T15:51:07.588 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.588 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [], 2024-08-19T15:51:07.588 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:51:07.588 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:51:07.588 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj14", 2024-08-19T15:51:07.588 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:51:07.588 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:51:07.588 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 2484217095, 2024-08-19T15:51:07.588 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:51:07.589 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:51:07.589 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:51:07.589 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.589 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'40", 2024-08-19T15:51:07.589 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'14", 2024-08-19T15:51:07.589 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4321.0:1", 2024-08-19T15:51:07.589 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 40, 2024-08-19T15:51:07.589 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.589 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:50:02.306313+0000", 2024-08-19T15:51:07.589 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:50:02.306884+0000", 2024-08-19T15:51:07.589 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:51:07.589 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:51:07.589 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:51:07.589 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:51:07.590 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.590 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:51:07.590 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:51:07.590 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0xc796de79", 2024-08-19T15:51:07.590 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:51:07.590 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:51:07.590 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:51:07.590 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:51:07.590 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:51:07.590 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:51:07.590 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.590 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:51:07.590 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.591 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:51:07.591 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.591 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:51:07.591 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:51:07.591 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.591 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.591 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:51:07.591 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:51:07.591 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:51:07.591 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.591 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:51:07.591 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1033, 2024-08-19T15:51:07.591 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:51:07.592 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:51:07.592 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:51:07.592 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.592 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.592 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.592 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.592 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.592 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.592 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:51:07.592 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:51:07.592 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.592 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.592 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:51:07.593 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:51:07.593 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:51:07.593 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.593 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:51:07.593 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:51:07.593 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:51:07.593 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:51:07.593 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:51:07.593 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.593 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.594 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.594 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.594 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.594 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.594 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.594 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.594 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:51:07.594 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj15", 2024-08-19T15:51:07.594 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:51:07.594 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:51:07.594 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:51:07.595 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 15 2024-08-19T15:51:07.595 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.595 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.595 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [ 2024-08-19T15:51:07.595 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_corrupted" 2024-08-19T15:51:07.595 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.595 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:51:07.595 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:51:07.595 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj15", 2024-08-19T15:51:07.595 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:51:07.595 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:51:07.595 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 612772309, 2024-08-19T15:51:07.595 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:51:07.596 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:51:07.596 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:51:07.596 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.596 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "16'15", 2024-08-19T15:51:07.596 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "0'0", 2024-08-19T15:51:07.596 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4273.0:1", 2024-08-19T15:51:07.596 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 15, 2024-08-19T15:51:07.596 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:51:07.596 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:49:59.260014+0000", 2024-08-19T15:51:07.596 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:49:59.260564+0000", 2024-08-19T15:51:07.596 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:51:07.596 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:51:07.596 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:51:07.596 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:51:07.597 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.597 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:51:07.597 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:51:07.597 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0x87f5ff96", 2024-08-19T15:51:07.597 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:51:07.597 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:51:07.597 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:51:07.597 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:51:07.597 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:51:07.597 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:51:07.597 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.597 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:51:07.597 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.597 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:51:07.598 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.598 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:51:07.598 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:51:07.598 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.598 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_corrupted" 2024-08-19T15:51:07.598 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.598 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:51:07.598 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": "Z2FyYmFnZQo=" 2024-08-19T15:51:07.598 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.598 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.598 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:51:07.598 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:51:07.599 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.599 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:51:07.599 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:51:07.599 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-08-19T15:51:07.599 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [] 2024-08-19T15:51:07.599 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.599 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.599 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.599 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.599 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.599 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:51:07.599 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj2", 2024-08-19T15:51:07.599 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:51:07.600 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:51:07.600 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:51:07.600 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 56 2024-08-19T15:51:07.600 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.600 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.600 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [ 2024-08-19T15:51:07.600 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_missing" 2024-08-19T15:51:07.600 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.600 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:51:07.600 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:51:07.600 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj2", 2024-08-19T15:51:07.600 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:51:07.600 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:51:07.600 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 1058988552, 2024-08-19T15:51:07.601 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:51:07.601 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:51:07.601 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:51:07.601 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.601 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "23'56", 2024-08-19T15:51:07.601 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "20'48", 2024-08-19T15:51:07.601 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4363.0:1", 2024-08-19T15:51:07.601 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 56, 2024-08-19T15:51:07.601 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 0, 2024-08-19T15:51:07.601 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:50:05.131215+0000", 2024-08-19T15:51:07.601 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:50:05.131755+0000", 2024-08-19T15:51:07.601 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:51:07.601 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:51:07.602 INFO:tasks.workunit.client.0.smithi102.stdout: "whiteout", 2024-08-19T15:51:07.602 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty" 2024-08-19T15:51:07.602 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.602 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:51:07.602 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:51:07.602 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0xffffffff", 2024-08-19T15:51:07.602 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:51:07.602 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:51:07.602 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:51:07.602 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:51:07.602 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:51:07.602 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:51:07.603 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.603 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:51:07.603 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.603 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:51:07.603 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.603 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:51:07.603 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:51:07.603 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.603 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_missing" 2024-08-19T15:51:07.603 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.603 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 0 2024-08-19T15:51:07.603 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.603 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.604 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:51:07.604 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:51:07.604 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.604 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 0, 2024-08-19T15:51:07.604 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:51:07.604 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 7, 2024-08-19T15:51:07.604 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:51:07.604 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.604 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 4, 2024-08-19T15:51:07.604 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:51:07.604 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:51:07.604 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:51:07.604 INFO:tasks.workunit.client.0.smithi102.stdout: 4, 2024-08-19T15:51:07.605 INFO:tasks.workunit.client.0.smithi102.stdout: 3, 2024-08-19T15:51:07.605 INFO:tasks.workunit.client.0.smithi102.stdout: 2, 2024-08-19T15:51:07.605 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:51:07.605 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.605 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.605 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.605 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 7, 2024-08-19T15:51:07.605 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1024, 2024-08-19T15:51:07.605 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:51:07.605 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:51:07.605 INFO:tasks.workunit.client.0.smithi102.stdout: 7, 2024-08-19T15:51:07.605 INFO:tasks.workunit.client.0.smithi102.stdout: 6, 2024-08-19T15:51:07.605 INFO:tasks.workunit.client.0.smithi102.stdout: 5 2024-08-19T15:51:07.606 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.606 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.606 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.606 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.606 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.606 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.606 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.606 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.606 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:51:07.606 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj3", 2024-08-19T15:51:07.606 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:51:07.606 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:51:07.606 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:51:07.607 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 44 2024-08-19T15:51:07.607 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.607 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.607 INFO:tasks.workunit.client.0.smithi102.stdout: "size_mismatch" 2024-08-19T15:51:07.607 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.607 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [ 2024-08-19T15:51:07.607 INFO:tasks.workunit.client.0.smithi102.stdout: "size_mismatch_info", 2024-08-19T15:51:07.607 INFO:tasks.workunit.client.0.smithi102.stdout: "obj_size_info_mismatch" 2024-08-19T15:51:07.607 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.607 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:51:07.608 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:51:07.608 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj3", 2024-08-19T15:51:07.608 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:51:07.608 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:51:07.608 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 1643547569, 2024-08-19T15:51:07.608 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:51:07.608 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:51:07.608 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:51:07.608 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.608 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "19'44", 2024-08-19T15:51:07.608 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "17'22", 2024-08-19T15:51:07.608 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4333.0:1", 2024-08-19T15:51:07.608 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 44, 2024-08-19T15:51:07.609 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 768, 2024-08-19T15:51:07.609 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:50:03.054545+0000", 2024-08-19T15:51:07.609 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:50:03.073422+0000", 2024-08-19T15:51:07.609 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:51:07.609 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:51:07.609 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:51:07.609 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:51:07.609 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.609 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:51:07.609 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:51:07.609 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0x3186b888", 2024-08-19T15:51:07.609 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:51:07.609 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:51:07.610 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:51:07.610 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:51:07.610 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:51:07.610 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:51:07.610 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.610 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:51:07.610 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.610 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:51:07.610 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.610 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:51:07.610 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:51:07.610 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: "size_mismatch_info", 2024-08-19T15:51:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: "obj_size_info_mismatch" 2024-08-19T15:51:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 3840, 2024-08-19T15:51:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: "object_info": { 2024-08-19T15:51:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:51:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj3", 2024-08-19T15:51:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:51:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:51:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 1643547569, 2024-08-19T15:51:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:51:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:51:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:51:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "19'44", 2024-08-19T15:51:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "17'22", 2024-08-19T15:51:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4333.0:1", 2024-08-19T15:51:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 44, 2024-08-19T15:51:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 768, 2024-08-19T15:51:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:50:03.054545+0000", 2024-08-19T15:51:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:50:03.073422+0000", 2024-08-19T15:51:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:51:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:51:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:51:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:51:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:51:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:51:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0x3186b888", 2024-08-19T15:51:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:51:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:51:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:51:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:51:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:51:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:51:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:51:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:51:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:51:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 768 2024-08-19T15:51:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:51:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj6", 2024-08-19T15:51:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:51:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:51:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:51:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 24 2024-08-19T15:51:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_inconsistency" 2024-08-19T15:51:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [], 2024-08-19T15:51:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:51:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:51:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj6", 2024-08-19T15:51:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:51:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:51:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 2202164420, 2024-08-19T15:51:07.616 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:51:07.616 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:51:07.616 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:51:07.616 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.616 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'24", 2024-08-19T15:51:07.616 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'6", 2024-08-19T15:51:07.616 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4297.0:1", 2024-08-19T15:51:07.616 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 24, 2024-08-19T15:51:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:50:01.168720+0000", 2024-08-19T15:51:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:50:01.169270+0000", 2024-08-19T15:51:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:51:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:51:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:51:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:51:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:51:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:51:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0xc796de79", 2024-08-19T15:51:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:51:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:51:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:51:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:51:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:51:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:51:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:51:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:51:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:51:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:51:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.619 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.619 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:51:07.619 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:51:07.619 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [] 2024-08-19T15:51:07.619 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.619 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.619 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.619 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:51:07.619 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:51:07.619 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.619 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.619 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:51:07.619 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:51:07.620 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:51:07.620 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.620 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:51:07.620 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:51:07.620 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:51:07.620 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:51:07.620 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:51:07.620 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.620 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.620 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.620 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.620 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.621 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.621 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.621 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.621 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:51:07.621 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj7", 2024-08-19T15:51:07.621 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:51:07.621 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:51:07.621 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:51:07.621 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 26 2024-08-19T15:51:07.621 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.621 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.621 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_inconsistency" 2024-08-19T15:51:07.621 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.621 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [], 2024-08-19T15:51:07.622 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:51:07.622 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:51:07.622 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj7", 2024-08-19T15:51:07.622 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:51:07.622 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:51:07.622 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 1552453721, 2024-08-19T15:51:07.622 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:51:07.622 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:51:07.622 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:51:07.622 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.622 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'26", 2024-08-19T15:51:07.622 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'7", 2024-08-19T15:51:07.622 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4300.0:1", 2024-08-19T15:51:07.623 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 26, 2024-08-19T15:51:07.623 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.623 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:50:01.318554+0000", 2024-08-19T15:51:07.623 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:50:01.319107+0000", 2024-08-19T15:51:07.623 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:51:07.623 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:51:07.623 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:51:07.623 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:51:07.623 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.623 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:51:07.623 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:51:07.623 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0xc796de79", 2024-08-19T15:51:07.623 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:51:07.624 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:51:07.624 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:51:07.624 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:51:07.624 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:51:07.624 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:51:07.624 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.624 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:51:07.624 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.624 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:51:07.624 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.624 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:51:07.624 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:51:07.624 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.625 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.625 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:51:07.625 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-08-19T15:51:07.625 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [] 2024-08-19T15:51:07.625 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.625 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.625 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.625 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:51:07.625 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:51:07.625 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.625 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.625 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:51:07.625 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:51:07.625 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:51:07.626 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.626 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:51:07.626 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:51:07.626 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:51:07.626 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:51:07.626 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:51:07.626 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.626 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.626 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.626 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.626 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.626 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.627 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.627 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.627 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:51:07.627 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj8", 2024-08-19T15:51:07.627 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:51:07.627 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:51:07.627 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:51:07.627 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 28 2024-08-19T15:51:07.627 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.627 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.627 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_inconsistency" 2024-08-19T15:51:07.627 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.627 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [], 2024-08-19T15:51:07.628 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:51:07.628 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:51:07.628 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj8", 2024-08-19T15:51:07.628 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:51:07.628 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:51:07.628 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 2381834917, 2024-08-19T15:51:07.628 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:51:07.628 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:51:07.628 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:51:07.628 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.628 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'28", 2024-08-19T15:51:07.628 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'8", 2024-08-19T15:51:07.629 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4303.0:1", 2024-08-19T15:51:07.629 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 28, 2024-08-19T15:51:07.629 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.629 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:50:01.452066+0000", 2024-08-19T15:51:07.629 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:50:01.452572+0000", 2024-08-19T15:51:07.629 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:51:07.629 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:51:07.629 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:51:07.629 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:51:07.629 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.629 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:51:07.629 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:51:07.630 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0xc796de79", 2024-08-19T15:51:07.630 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:51:07.630 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:51:07.630 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:51:07.630 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:51:07.630 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:51:07.630 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:51:07.630 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.630 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:51:07.630 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.630 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:51:07.630 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.630 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:51:07.631 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:51:07.631 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.631 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.631 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:51:07.631 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-08-19T15:51:07.631 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:51:07.631 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.631 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:51:07.631 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:51:07.631 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:51:07.631 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:51:07.631 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:51:07.631 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.632 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.632 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.632 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.632 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.632 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.632 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:51:07.632 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:51:07.632 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.632 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.632 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:51:07.632 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:51:07.632 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:51:07.633 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.633 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:51:07.633 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:51:07.633 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:51:07.633 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:51:07.633 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:51:07.633 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.633 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.633 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.633 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.633 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.633 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.633 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.634 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.634 INFO:tasks.workunit.client.0.smithi102.stdout: "object": { 2024-08-19T15:51:07.634 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "obj9", 2024-08-19T15:51:07.634 INFO:tasks.workunit.client.0.smithi102.stdout: "nspace": "", 2024-08-19T15:51:07.634 INFO:tasks.workunit.client.0.smithi102.stdout: "locator": "", 2024-08-19T15:51:07.634 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": "head", 2024-08-19T15:51:07.634 INFO:tasks.workunit.client.0.smithi102.stdout: "version": 30 2024-08-19T15:51:07.634 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.634 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [ 2024-08-19T15:51:07.634 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset_inconsistency" 2024-08-19T15:51:07.634 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.634 INFO:tasks.workunit.client.0.smithi102.stdout: "union_shard_errors": [], 2024-08-19T15:51:07.635 INFO:tasks.workunit.client.0.smithi102.stdout: "selected_object_info": { 2024-08-19T15:51:07.635 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": { 2024-08-19T15:51:07.635 INFO:tasks.workunit.client.0.smithi102.stdout: "oid": "obj9", 2024-08-19T15:51:07.635 INFO:tasks.workunit.client.0.smithi102.stdout: "key": "", 2024-08-19T15:51:07.635 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": -2, 2024-08-19T15:51:07.635 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": 3833113727, 2024-08-19T15:51:07.635 INFO:tasks.workunit.client.0.smithi102.stdout: "max": 0, 2024-08-19T15:51:07.635 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-08-19T15:51:07.635 INFO:tasks.workunit.client.0.smithi102.stdout: "namespace": "" 2024-08-19T15:51:07.635 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.635 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "17'30", 2024-08-19T15:51:07.635 INFO:tasks.workunit.client.0.smithi102.stdout: "prior_version": "16'9", 2024-08-19T15:51:07.635 INFO:tasks.workunit.client.0.smithi102.stdout: "last_reqid": "client.4306.0:1", 2024-08-19T15:51:07.635 INFO:tasks.workunit.client.0.smithi102.stdout: "user_version": 30, 2024-08-19T15:51:07.636 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.636 INFO:tasks.workunit.client.0.smithi102.stdout: "mtime": "2024-08-19T15:50:01.594689+0000", 2024-08-19T15:51:07.636 INFO:tasks.workunit.client.0.smithi102.stdout: "local_mtime": "2024-08-19T15:50:01.595202+0000", 2024-08-19T15:51:07.636 INFO:tasks.workunit.client.0.smithi102.stdout: "lost": 0, 2024-08-19T15:51:07.636 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": [ 2024-08-19T15:51:07.636 INFO:tasks.workunit.client.0.smithi102.stdout: "dirty", 2024-08-19T15:51:07.636 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest" 2024-08-19T15:51:07.636 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T15:51:07.636 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_seq": 0, 2024-08-19T15:51:07.636 INFO:tasks.workunit.client.0.smithi102.stdout: "truncate_size": 0, 2024-08-19T15:51:07.636 INFO:tasks.workunit.client.0.smithi102.stdout: "data_digest": "0xc796de79", 2024-08-19T15:51:07.636 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_digest": "0xffffffff", 2024-08-19T15:51:07.637 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_object_size": 0, 2024-08-19T15:51:07.637 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_write_size": 0, 2024-08-19T15:51:07.637 INFO:tasks.workunit.client.0.smithi102.stdout: "alloc_hint_flags": 0, 2024-08-19T15:51:07.637 INFO:tasks.workunit.client.0.smithi102.stdout: "manifest": { 2024-08-19T15:51:07.637 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 0 2024-08-19T15:51:07.637 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.637 INFO:tasks.workunit.client.0.smithi102.stdout: "watchers": {} 2024-08-19T15:51:07.637 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.637 INFO:tasks.workunit.client.0.smithi102.stdout: "shards": [ 2024-08-19T15:51:07.637 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.637 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-08-19T15:51:07.637 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": false, 2024-08-19T15:51:07.637 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.638 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.638 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:51:07.638 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:51:07.638 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:51:07.638 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.638 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:51:07.638 INFO:tasks.workunit.client.0.smithi102.stdout: "size": "????", 2024-08-19T15:51:07.638 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:51:07.638 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:51:07.638 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:51:07.638 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.638 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.639 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.639 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.639 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T15:51:07.639 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.639 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-08-19T15:51:07.639 INFO:tasks.workunit.client.0.smithi102.stdout: "primary": true, 2024-08-19T15:51:07.639 INFO:tasks.workunit.client.0.smithi102.stdout: "errors": [], 2024-08-19T15:51:07.639 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 256, 2024-08-19T15:51:07.639 INFO:tasks.workunit.client.0.smithi102.stdout: "snapset": { 2024-08-19T15:51:07.639 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 1, 2024-08-19T15:51:07.639 INFO:tasks.workunit.client.0.smithi102.stdout: "clones": [ 2024-08-19T15:51:07.639 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T15:51:07.639 INFO:tasks.workunit.client.0.smithi102.stdout: "snap": 1, 2024-08-19T15:51:07.639 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 1032, 2024-08-19T15:51:07.639 INFO:tasks.workunit.client.0.smithi102.stdout: "overlap": "[]", 2024-08-19T15:51:07.640 INFO:tasks.workunit.client.0.smithi102.stdout: "snaps": [ 2024-08-19T15:51:07.640 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-08-19T15:51:07.640 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.640 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.640 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.640 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.640 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.640 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.640 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T15:51:07.640 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T15:51:07.640 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:51:07.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:797: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2024-08-19T15:51:07.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:800: _scrub_snaps_multi: '[' replica = replica ']' 2024-08-19T15:51:07.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:802: _scrub_snaps_multi: scruberrors=20 2024-08-19T15:51:07.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:803: _scrub_snaps_multi: jq .inconsistents 2024-08-19T15:51:07.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:803: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-08-19T15:51:07.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2024-08-19T15:51:07.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-08-19T15:51:07.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1067: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-08-19T15:51:07.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-08-19T15:51:07.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1068: _scrub_snaps_multi: test no = yes 2024-08-19T15:51:07.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1073: _scrub_snaps_multi: test '' = yes 2024-08-19T15:51:07.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: find td/osd-scrub-snaps 2024-08-19T15:51:07.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2024-08-19T15:51:07.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.1.pid 2024-08-19T15:51:07.644 INFO:tasks.workunit.client.0.smithi102.stderr:td/osd-scrub-snaps/osd.0.pid' 2024-08-19T15:51:07.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids= 2024-08-19T15:51:07.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-08-19T15:51:07.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2024-08-19T15:51:07.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='534876 ' 2024-08-19T15:51:07.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-08-19T15:51:07.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2024-08-19T15:51:07.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='534336 ' 2024-08-19T15:51:07.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1085: _scrub_snaps_multi: ERRORS=0 2024-08-19T15:51:07.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1089: _scrub_snaps_multi: '[' replica = primary ']' 2024-08-19T15:51:07.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1111: _scrub_snaps_multi: ceph pg dump pgs 2024-08-19T15:51:07.881 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-19T15:51:07.882 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent 2024-08-19T15:51:01.469281+0000 23'56 30:102 [1,0] 1 [1,0] 1 23'56 2024-08-19T15:51:01.469199+0000 0'0 2024-08-19T15:49:50.948603+0000 0 1 periodic scrub scheduled @ 2024-08-20T23:54:33.031869+0000 36 0 2024-08-19T15:51:07.882 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-08-19T15:51:07.882 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-19T15:51:07.882 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:51:07.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2024-08-19T15:51:07.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 534876 2024-08-19T15:51:07.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2024-08-19T15:51:07.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 534336 2024-08-19T15:51:07.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1122: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2024-08-19T15:51:07.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:51:07.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:51:07.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:51:07.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:51:07.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:51:13.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:51:13.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1124: _scrub_snaps_multi: declare -a err_strings 2024-08-19T15:51:13.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1125: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2024-08-19T15:51:13.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1126: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2024-08-19T15:51:13.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2024-08-19T15:51:13.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2024-08-19T15:51:13.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2024-08-19T15:51:13.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1130: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2024-08-19T15:51:13.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2024-08-19T15:51:13.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1132: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2024-08-19T15:51:13.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-08-19T15:51:13.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2024-08-19T15:51:13.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-08-19T15:51:13.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-08-19T15:51:13.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-08-19T15:51:13.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2024-08-19T15:51:13.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-08-19T15:51:13.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2024-08-19T15:51:13.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-08-19T15:51:13.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-08-19T15:51:13.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-08-19T15:51:13.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2024-08-19T15:51:13.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-08-19T15:51:13.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2024-08-19T15:51:13.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-08-19T15:51:13.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.1.log 2024-08-19T15:51:13.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1144: _scrub_snaps_multi: declare -a rep_err_strings 2024-08-19T15:51:13.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: eval echo '$replica' 2024-08-19T15:51:13.229 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: echo 0 2024-08-19T15:51:13.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: osd=0 2024-08-19T15:51:13.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-08-19T15:51:13.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1147: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2024-08-19T15:51:13.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1149: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2024-08-19T15:51:13.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1156: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2024-08-19T15:51:13.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1162: _scrub_snaps_multi: echo 'TEST PASSED' 2024-08-19T15:51:13.232 INFO:tasks.workunit.client.0.smithi102.stdout:TEST PASSED 2024-08-19T15:51:13.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1163: _scrub_snaps_multi: return 0 2024-08-19T15:51:13.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1171: TEST_scrub_snaps_replica: err=0 2024-08-19T15:51:13.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1172: TEST_scrub_snaps_replica: CEPH_ARGS='--fsid=7a00e832-2980-48a9-94c1-268d827f78a0 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-08-19T15:51:13.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1173: TEST_scrub_snaps_replica: return 0 2024-08-19T15:51:13.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-08-19T15:51:13.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-08-19T15:51:13.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:51:13.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-08-19T15:51:13.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:51:13.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:51:13.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:51:13.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:51:13.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:51:13.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:51:13.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:51:13.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:51:13.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:51:13.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:51:13.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:51:13.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:51:13.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:51:13.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:51:13.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:51:13.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:51:13.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:51:13.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:51:13.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-08-19T15:51:13.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:51:13.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:51:13.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:51:13.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.470418 2024-08-19T15:51:13.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:51:13.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:51:13.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-08-19T15:51:13.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-snaps 0 2024-08-19T15:51:13.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-08-19T15:51:13.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-08-19T15:51:13.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-08-19T15:51:13.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:51:13.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:51:13.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:51:13.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:51:13.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:51:13.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:51:13.265 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:51:13.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:51:13.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:51:13.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:51:13.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:51:13.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:51:13.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:51:13.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:51:13.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:51:13.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:51:13.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:51:13.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-08-19T15:51:13.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-08-19T15:51:13.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:51:13.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:51:13.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470418 2024-08-19T15:51:13.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.470418 2024-08-19T15:51:13.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:51:13.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:51:13.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-08-19T15:51:13.276 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-19T15:51:13.276 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-19T15:51:13.306 INFO:tasks.workunit:Running workunit scrub/osd-scrub-test.sh... 2024-08-19T15:51:13.306 DEBUG:teuthology.orchestra.run.smithi102:workunit test scrub/osd-scrub-test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=78215f5a1cf5eaf321d506a66539942e12851f34 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh 2024-08-19T15:51:13.372 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-19T15:51:13.379 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-19T15:51:13.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-08-19T15:51:13.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-08-19T15:51:13.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-08-19T15:51:13.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-08-19T15:51:13.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-08-19T15:51:13.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-08-19T15:51:13.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-08-19T15:51:13.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-08-19T15:51:13.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-test 2024-08-19T15:51:13.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:21: run: local dir=td/osd-scrub-test 2024-08-19T15:51:13.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:22: run: shift 2024-08-19T15:51:13.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: export CEPH_MON=127.0.0.1:7138 2024-08-19T15:51:13.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: CEPH_MON=127.0.0.1:7138 2024-08-19T15:51:13.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:25: run: export CEPH_ARGS 2024-08-19T15:51:13.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: uuidgen 2024-08-19T15:51:13.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: CEPH_ARGS+='--fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none ' 2024-08-19T15:51:13.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7138 ' 2024-08-19T15:51:13.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-08-19T15:51:13.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: set 2024-08-19T15:51:13.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-08-19T15:51:13.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: local 'funcs=TEST_deep_scrub_abort 2024-08-19T15:51:13.383 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_dump_scrub_schedule 2024-08-19T15:51:13.383 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_interval_changes 2024-08-19T15:51:13.383 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_just_deep_scrubs 2024-08-19T15:51:13.383 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_pg_dump_objects_scrubbed 2024-08-19T15:51:13.384 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_scrub_abort 2024-08-19T15:51:13.384 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_scrub_extended_sleep 2024-08-19T15:51:13.384 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_scrub_permit_time 2024-08-19T15:51:13.384 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_scrub_test' 2024-08-19T15:51:13.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-08-19T15:51:13.384 INFO:tasks.workunit.client.0.smithi102.stdout:-------------- Prepare Test TEST_deep_scrub_abort ------------------- 2024-08-19T15:51:13.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_deep_scrub_abort -------------------' 2024-08-19T15:51:13.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-08-19T15:51:13.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-08-19T15:51:13.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-08-19T15:51:13.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-08-19T15:51:13.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T15:51:13.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-08-19T15:51:13.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T15:51:13.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T15:51:13.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T15:51:13.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T15:51:13.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T15:51:13.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T15:51:13.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T15:51:13.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T15:51:13.388 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T15:51:13.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T15:51:13.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T15:51:13.388 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T15:51:13.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:51:13.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T15:51:13.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T15:51:13.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T15:51:13.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:51:13.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T15:51:13.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-08-19T15:51:13.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T15:51:13.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:51:13.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T15:51:13.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.536210 2024-08-19T15:51:13.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T15:51:13.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T15:51:13.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-08-19T15:51:13.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T15:51:13.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:51:13.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T15:51:13.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.536210 2024-08-19T15:51:13.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T15:51:13.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-19T15:51:13.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-19T15:51:13.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T15:51:13.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-08-19T15:51:13.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_deep_scrub_abort -----------------------' 2024-08-19T15:51:13.399 INFO:tasks.workunit.client.0.smithi102.stdout:-------------- Run Test TEST_deep_scrub_abort ----------------------- 2024-08-19T15:51:13.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_deep_scrub_abort td/osd-scrub-test 2024-08-19T15:51:13.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:414: TEST_deep_scrub_abort: local dir=td/osd-scrub-test 2024-08-19T15:51:13.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:415: TEST_deep_scrub_abort: _scrub_abort td/osd-scrub-test deep-scrub 2024-08-19T15:51:13.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:298: _scrub_abort: local dir=td/osd-scrub-test 2024-08-19T15:51:13.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:299: _scrub_abort: local poolname=test 2024-08-19T15:51:13.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:300: _scrub_abort: local OSDS=3 2024-08-19T15:51:13.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:301: _scrub_abort: local objects=1000 2024-08-19T15:51:13.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:302: _scrub_abort: local type=deep-scrub 2024-08-19T15:51:13.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:304: _scrub_abort: TESTDATA=testdata.536210 2024-08-19T15:51:13.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:305: _scrub_abort: test deep-scrub = scrub 2024-08-19T15:51:13.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:310: _scrub_abort: stopscrub=nodeep-scrub 2024-08-19T15:51:13.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: check=nodeep_scrub 2024-08-19T15:51:13.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:314: _scrub_abort: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-08-19T15:51:13.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-test 2024-08-19T15:51:13.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T15:51:13.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T15:51:13.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T15:51:13.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-test/a 2024-08-19T15:51:13.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-08-19T15:51:13.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T15:51:13.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:51:13.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:51:13.784 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:51:13.784 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:51:13.784 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T15:51:13.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T15:51:13.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-08-19T15:51:13.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T15:51:13.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T15:51:13.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:51:13.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:51:13.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T15:51:13.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T15:51:13.969 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:51:13.969 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:51:13.969 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:51:13.969 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:51:13.969 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:51:13.969 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T15:51:13.969 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.536210/ceph-mon.a.asok 2024-08-19T15:51:13.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:51:13.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.536210/ceph-mon.a.asok config get fsid 2024-08-19T15:51:14.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T15:51:14.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T15:51:14.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T15:51:14.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T15:51:14.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T15:51:14.052 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T15:51:14.052 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T15:51:14.052 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T15:51:14.052 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T15:51:14.052 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:51:14.052 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T15:51:14.052 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.536210/ceph-mon.a.asok 2024-08-19T15:51:14.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T15:51:14.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.536210/ceph-mon.a.asok config get mon_host 2024-08-19T15:51:14.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: run_mgr td/osd-scrub-test x 2024-08-19T15:51:14.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-test 2024-08-19T15:51:14.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T15:51:14.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T15:51:14.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T15:51:14.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-test/x 2024-08-19T15:51:14.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T15:51:14.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T15:51:14.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:51:14.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:51:14.938 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:51:14.938 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:51:14.938 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T15:51:14.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T15:51:14.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:51:14.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T15:51:14.964 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:316: _scrub_abort: expr 3 - 1 2024-08-19T15:51:14.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:316: _scrub_abort: seq 0 2 2024-08-19T15:51:14.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:316: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:51:14.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:322: _scrub_abort: run_osd td/osd-scrub-test 0 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-08-19T15:51:14.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-08-19T15:51:14.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:51:14.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T15:51:14.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:51:14.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/0 2024-08-19T15:51:14.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-08-19T15:51:14.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:51:14.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:51:14.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:51:14.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-08-19T15:51:14.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-08-19T15:51:14.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:51:14.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:51:14.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-08-19T15:51:14.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:51:14.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:51:14.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:51:14.972 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:51:14.972 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:51:14.972 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T15:51:14.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T15:51:14.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T15:51:14.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:51:14.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:51:14.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:51:14.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-08-19T15:51:14.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-08-19T15:51:14.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:51:14.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:51:14.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:51:14.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:51:14.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:51:14.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-08-19T15:51:14.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/0 2024-08-19T15:51:14.974 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:51:14.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1412d074-0b16-4b02-804d-f344a50c8849 2024-08-19T15:51:14.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1412d074-0b16-4b02-804d-f344a50c8849' 2024-08-19T15:51:14.974 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 1412d074-0b16-4b02-804d-f344a50c8849 2024-08-19T15:51:14.974 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:51:14.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDyacNmlKa0OhAAY+zblmfCEiwlZzFbqtiCJw== 2024-08-19T15:51:14.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDyacNmlKa0OhAAY+zblmfCEiwlZzFbqtiCJw=="}' 2024-08-19T15:51:14.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1412d074-0b16-4b02-804d-f344a50c8849 -i td/osd-scrub-test/0/new.json 2024-08-19T15:51:15.185 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:51:15.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/0/new.json 2024-08-19T15:51:15.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQDyacNmlKa0OhAAY+zblmfCEiwlZzFbqtiCJw== --osd-uuid 1412d074-0b16-4b02-804d-f344a50c8849 2024-08-19T15:51:15.225 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:15.224+0000 7f0465d88640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:51:15.236 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:15.235+0000 7f0465d88640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:51:15.237 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:15.236+0000 7f0465d88640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:51:15.238 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:15.236+0000 7f0465d88640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-08-19T15:51:15.238 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:15.236+0000 7f0465d88640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-08-19T15:51:15.238 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:15.236+0000 7f0465d88640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-08-19T15:51:15.320 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:15.318+0000 7f0465d88640 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-08-19T15:51:20.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-08-19T15:51:20.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:51:20.836 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T15:51:20.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T15:51:20.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:51:21.180 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T15:51:21.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T15:51:21.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-08-19T15:51:21.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:51:21.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:51:21.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:51:21.209 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:21.208+0000 7f0935f40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:51:21.225 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:21.223+0000 7f0935f40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:51:21.226 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:21.224+0000 7f0935f40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:51:21.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T15:51:21.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:51:21.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T15:51:21.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:51:21.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:51:21.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:51:21.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:51:21.485 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:51:21.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:51:21.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:51:21.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:51:21.770 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:21.769+0000 7f0935f40640 -1 Falling back to public interface 2024-08-19T15:51:22.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:51:22.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:51:22.748 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:51:22.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:51:22.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:51:22.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:51:23.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:51:24.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:51:24.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:51:24.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:51:24.022 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:51:24.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:51:24.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:51:24.085 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:24.083+0000 7f0935f40640 -1 osd.0 0 log_to_monitors true 2024-08-19T15:51:24.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:51:25.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:51:25.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:51:25.321 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:51:25.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:51:25.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:51:25.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T15:51:25.594 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3186498073,v1:127.0.0.1:6803/3186498073] [v2:127.0.0.1:6804/3186498073,v1:127.0.0.1:6805/3186498073] exists,up 1412d074-0b16-4b02-804d-f344a50c8849 2024-08-19T15:51:25.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:51:25.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:51:25.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:51:25.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:316: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:51:25.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:322: _scrub_abort: run_osd td/osd-scrub-test 1 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-08-19T15:51:25.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-08-19T15:51:25.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:51:25.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T15:51:25.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:51:25.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/1 2024-08-19T15:51:25.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-08-19T15:51:25.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:51:25.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:51:25.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:51:25.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-08-19T15:51:25.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-08-19T15:51:25.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:51:25.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:51:25.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-08-19T15:51:25.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:51:25.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:51:25.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:51:25.600 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:51:25.600 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:51:25.600 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T15:51:25.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T15:51:25.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T15:51:25.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:51:25.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:51:25.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:51:25.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-08-19T15:51:25.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-08-19T15:51:25.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:51:25.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:51:25.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:51:25.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:51:25.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:51:25.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-08-19T15:51:25.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/1 2024-08-19T15:51:25.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:51:25.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=65394ca7-d5b2-4ee4-b9b1-bca5824d9ac1 2024-08-19T15:51:25.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 65394ca7-d5b2-4ee4-b9b1-bca5824d9ac1' 2024-08-19T15:51:25.601 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 65394ca7-d5b2-4ee4-b9b1-bca5824d9ac1 2024-08-19T15:51:25.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:51:25.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD9acNm0hmHJBAAVpq1xmD5hnByPUYEcl2ADw== 2024-08-19T15:51:25.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD9acNm0hmHJBAAVpq1xmD5hnByPUYEcl2ADw=="}' 2024-08-19T15:51:25.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 65394ca7-d5b2-4ee4-b9b1-bca5824d9ac1 -i td/osd-scrub-test/1/new.json 2024-08-19T15:51:25.963 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:51:25.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/1/new.json 2024-08-19T15:51:25.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQD9acNm0hmHJBAAVpq1xmD5hnByPUYEcl2ADw== --osd-uuid 65394ca7-d5b2-4ee4-b9b1-bca5824d9ac1 2024-08-19T15:51:26.005 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:26.004+0000 7f39e1c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:51:26.031 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:26.029+0000 7f39e1c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:51:26.032 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:26.030+0000 7f39e1c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:51:26.032 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:26.031+0000 7f39e1c16640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-08-19T15:51:26.032 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:26.031+0000 7f39e1c16640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-08-19T15:51:26.033 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:26.031+0000 7f39e1c16640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-08-19T15:51:26.098 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:26.096+0000 7f39e1c16640 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-08-19T15:51:31.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-08-19T15:51:31.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:51:31.934 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T15:51:31.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T15:51:31.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:51:32.280 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T15:51:32.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T15:51:32.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-08-19T15:51:32.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:51:32.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:51:32.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:51:32.308 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:32.307+0000 7fe7183fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:51:32.310 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:32.309+0000 7fe7183fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:51:32.311 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:32.310+0000 7fe7183fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:51:32.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T15:51:32.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:51:32.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T15:51:32.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:51:32.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:51:32.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:51:32.545 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:51:32.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:51:32.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:51:32.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:51:32.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:51:32.843 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:32.842+0000 7fe7183fa640 -1 Falling back to public interface 2024-08-19T15:51:33.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:51:33.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:51:33.809 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:51:33.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:51:33.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:51:33.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:51:34.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:51:35.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:51:35.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:51:35.078 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:51:35.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:51:35.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:51:35.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:51:35.322 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:35.320+0000 7fe7183fa640 -1 osd.1 0 log_to_monitors true 2024-08-19T15:51:35.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:51:36.445 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:51:36.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:51:36.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:51:36.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:51:36.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:51:36.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T15:51:36.741 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1751944331,v1:127.0.0.1:6811/1751944331] [v2:127.0.0.1:6812/1751944331,v1:127.0.0.1:6813/1751944331] exists,up 65394ca7-d5b2-4ee4-b9b1-bca5824d9ac1 2024-08-19T15:51:36.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:51:36.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:51:36.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:51:36.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:316: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T15:51:36.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:322: _scrub_abort: run_osd td/osd-scrub-test 2 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-08-19T15:51:36.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-08-19T15:51:36.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T15:51:36.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T15:51:36.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T15:51:36.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/2 2024-08-19T15:51:36.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-08-19T15:51:36.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T15:51:36.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T15:51:36.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T15:51:36.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-08-19T15:51:36.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-08-19T15:51:36.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T15:51:36.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T15:51:36.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-08-19T15:51:36.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T15:51:36.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T15:51:36.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T15:51:36.747 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T15:51:36.747 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T15:51:36.747 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T15:51:36.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T15:51:36.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T15:51:36.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T15:51:36.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T15:51:36.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T15:51:36.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-08-19T15:51:36.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-08-19T15:51:36.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T15:51:36.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T15:51:36.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T15:51:36.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T15:51:36.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T15:51:36.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-08-19T15:51:36.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/2 2024-08-19T15:51:36.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T15:51:36.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fe61de50-3cc7-4dc3-9b1d-68e9aac5f459 2024-08-19T15:51:36.749 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 fe61de50-3cc7-4dc3-9b1d-68e9aac5f459 2024-08-19T15:51:36.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 fe61de50-3cc7-4dc3-9b1d-68e9aac5f459' 2024-08-19T15:51:36.750 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T15:51:36.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAIasNmbRhXLRAAsLL/MQv0Fs/52VKazXqoCQ== 2024-08-19T15:51:36.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAIasNmbRhXLRAAsLL/MQv0Fs/52VKazXqoCQ=="}' 2024-08-19T15:51:36.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fe61de50-3cc7-4dc3-9b1d-68e9aac5f459 -i td/osd-scrub-test/2/new.json 2024-08-19T15:51:37.159 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:51:37.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/2/new.json 2024-08-19T15:51:37.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQAIasNmbRhXLRAAsLL/MQv0Fs/52VKazXqoCQ== --osd-uuid fe61de50-3cc7-4dc3-9b1d-68e9aac5f459 2024-08-19T15:51:37.200 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:37.199+0000 7f6b4b213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:51:37.244 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:37.242+0000 7f6b4b213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:51:37.244 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:37.243+0000 7f6b4b213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:51:37.245 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:37.244+0000 7f6b4b213640 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-08-19T15:51:37.245 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:37.244+0000 7f6b4b213640 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-08-19T15:51:37.245 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:37.244+0000 7f6b4b213640 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-08-19T15:51:37.320 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:37.319+0000 7f6b4b213640 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-08-19T15:51:43.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-08-19T15:51:43.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T15:51:43.055 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T15:51:43.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T15:51:43.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T15:51:43.402 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T15:51:43.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T15:51:43.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-08-19T15:51:43.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T15:51:43.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T15:51:43.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T15:51:43.430 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:43.428+0000 7f804da82640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:51:43.432 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:43.430+0000 7f804da82640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:51:43.433 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:43.431+0000 7f804da82640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T15:51:43.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T15:51:43.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T15:51:43.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T15:51:43.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T15:51:43.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T15:51:43.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:51:43.667 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T15:51:43.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T15:51:43.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:51:43.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:51:43.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:51:43.954 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:43.953+0000 7f804da82640 -1 Falling back to public interface 2024-08-19T15:51:44.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:51:44.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:51:44.931 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T15:51:44.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T15:51:44.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:51:44.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:51:45.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:51:46.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:51:46.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:51:46.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T15:51:46.206 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T15:51:46.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:51:46.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:51:46.267 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T15:51:46.266+0000 7f804da82640 -1 osd.2 0 log_to_monitors true 2024-08-19T15:51:46.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T15:51:47.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T15:51:47.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T15:51:47.483 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T15:51:47.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T15:51:47.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T15:51:47.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T15:51:47.794 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3904778156,v1:127.0.0.1:6819/3904778156] [v2:127.0.0.1:6820/3904778156,v1:127.0.0.1:6821/3904778156] exists,up fe61de50-3cc7-4dc3-9b1d-68e9aac5f459 2024-08-19T15:51:47.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T15:51:47.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T15:51:47.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T15:51:47.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: create_pool test 1 1 2024-08-19T15:51:47.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-08-19T15:51:48.163 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' created 2024-08-19T15:51:48.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T15:51:49.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:331: _scrub_abort: wait_for_clean 2024-08-19T15:51:49.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T15:51:49.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T15:51:49.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T15:51:49.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T15:51:49.177 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T15:51:49.178 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T15:51:49.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T15:51:49.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T15:51:49.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T15:51:49.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T15:51:49.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T15:51:49.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T15:51:49.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T15:51:49.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:51:49.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:51:49.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:51:49.566 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:51:49.566 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:51:49.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:51:49.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:51:49.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:51:49.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836486 2024-08-19T15:51:49.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836486 2024-08-19T15:51:49.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486' 2024-08-19T15:51:49.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:51:49.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:51:49.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-08-19T15:51:49.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-08-19T15:51:49.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486 1-42949672964' 2024-08-19T15:51:49.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:51:49.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:51:49.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509442 2024-08-19T15:51:49.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509442 2024-08-19T15:51:49.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486 1-42949672964 2-64424509442' 2024-08-19T15:51:49.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:51:49.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836486 2024-08-19T15:51:49.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:51:49.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:51:49.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836486 2024-08-19T15:51:49.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:51:49.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836486 2024-08-19T15:51:49.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-08-19T15:51:49.994 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836486 2024-08-19T15:51:49.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:51:50.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836486 2024-08-19T15:51:50.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:51:51.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:51:51.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:51:51.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836486 2024-08-19T15:51:51.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:51:51.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-08-19T15:51:51.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:51:51.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:51:51.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-08-19T15:51:51.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:51:51.534 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672964 2024-08-19T15:51:51.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-08-19T15:51:51.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-08-19T15:51:51.534 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:51:51.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672964 2024-08-19T15:51:51.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:51:51.802 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509442 2024-08-19T15:51:51.802 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:51:51.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:51:51.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509442 2024-08-19T15:51:51.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:51:51.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509442 2024-08-19T15:51:51.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-08-19T15:51:51.805 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509442 2024-08-19T15:51:51.806 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:51:52.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509442 -lt 64424509442 2024-08-19T15:51:52.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T15:51:52.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:51:52.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:51:52.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T15:51:52.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T15:51:52.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T15:51:52.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T15:51:52.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T15:51:52.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T15:51:52.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T15:51:52.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T15:51:52.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T15:51:52.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T15:51:52.698 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T15:51:52.698 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T15:51:53.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T15:51:53.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T15:51:53.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T15:51:53.024 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: ceph osd dump 2024-08-19T15:51:53.024 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: grep '^pool.*['\'']test['\'']' 2024-08-19T15:51:53.024 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: awk '{ print $2 }' 2024-08-19T15:51:53.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: poolid=1 2024-08-19T15:51:53.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: dd if=/dev/urandom of=testdata.536210 bs=1032 count=1 2024-08-19T15:51:53.302 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records in 2024-08-19T15:51:53.302 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records out 2024-08-19T15:51:53.302 INFO:tasks.workunit.client.0.smithi102.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.5574e-05 s, 22.6 MB/s 2024-08-19T15:51:53.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: seq 1 1000 2024-08-19T15:51:53.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:53.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj1 testdata.536210 2024-08-19T15:51:53.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:53.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj2 testdata.536210 2024-08-19T15:51:53.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:53.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj3 testdata.536210 2024-08-19T15:51:53.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:53.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj4 testdata.536210 2024-08-19T15:51:53.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:53.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj5 testdata.536210 2024-08-19T15:51:54.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:54.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj6 testdata.536210 2024-08-19T15:51:54.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:54.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj7 testdata.536210 2024-08-19T15:51:54.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:54.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj8 testdata.536210 2024-08-19T15:51:54.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:54.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj9 testdata.536210 2024-08-19T15:51:54.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:54.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj10 testdata.536210 2024-08-19T15:51:54.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:54.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj11 testdata.536210 2024-08-19T15:51:54.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:54.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj12 testdata.536210 2024-08-19T15:51:55.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:55.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj13 testdata.536210 2024-08-19T15:51:55.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:55.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj14 testdata.536210 2024-08-19T15:51:55.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:55.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj15 testdata.536210 2024-08-19T15:51:55.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:55.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj16 testdata.536210 2024-08-19T15:51:55.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:55.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj17 testdata.536210 2024-08-19T15:51:55.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:55.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj18 testdata.536210 2024-08-19T15:51:55.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:55.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj19 testdata.536210 2024-08-19T15:51:56.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:56.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj20 testdata.536210 2024-08-19T15:51:56.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:56.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj21 testdata.536210 2024-08-19T15:51:56.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:56.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj22 testdata.536210 2024-08-19T15:51:56.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:56.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj23 testdata.536210 2024-08-19T15:51:56.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:56.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj24 testdata.536210 2024-08-19T15:51:56.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:56.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj25 testdata.536210 2024-08-19T15:51:56.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:56.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj26 testdata.536210 2024-08-19T15:51:57.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:57.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj27 testdata.536210 2024-08-19T15:51:57.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:57.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj28 testdata.536210 2024-08-19T15:51:57.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:57.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj29 testdata.536210 2024-08-19T15:51:57.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:57.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj30 testdata.536210 2024-08-19T15:51:57.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:57.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj31 testdata.536210 2024-08-19T15:51:57.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:57.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj32 testdata.536210 2024-08-19T15:51:57.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:57.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj33 testdata.536210 2024-08-19T15:51:58.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:58.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj34 testdata.536210 2024-08-19T15:51:58.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:58.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj35 testdata.536210 2024-08-19T15:51:58.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:58.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj36 testdata.536210 2024-08-19T15:51:58.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:58.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj37 testdata.536210 2024-08-19T15:51:58.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:58.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj38 testdata.536210 2024-08-19T15:51:58.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:58.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj39 testdata.536210 2024-08-19T15:51:58.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:58.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj40 testdata.536210 2024-08-19T15:51:59.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:59.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj41 testdata.536210 2024-08-19T15:51:59.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:59.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj42 testdata.536210 2024-08-19T15:51:59.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:59.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj43 testdata.536210 2024-08-19T15:51:59.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:59.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj44 testdata.536210 2024-08-19T15:51:59.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:59.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj45 testdata.536210 2024-08-19T15:51:59.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:59.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj46 testdata.536210 2024-08-19T15:51:59.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:51:59.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj47 testdata.536210 2024-08-19T15:52:00.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:00.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj48 testdata.536210 2024-08-19T15:52:00.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:00.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj49 testdata.536210 2024-08-19T15:52:00.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:00.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj50 testdata.536210 2024-08-19T15:52:00.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:00.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj51 testdata.536210 2024-08-19T15:52:00.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:00.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj52 testdata.536210 2024-08-19T15:52:00.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:00.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj53 testdata.536210 2024-08-19T15:52:00.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:00.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj54 testdata.536210 2024-08-19T15:52:00.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:00.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj55 testdata.536210 2024-08-19T15:52:01.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:01.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj56 testdata.536210 2024-08-19T15:52:01.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:01.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj57 testdata.536210 2024-08-19T15:52:01.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:01.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj58 testdata.536210 2024-08-19T15:52:01.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:01.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj59 testdata.536210 2024-08-19T15:52:01.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:01.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj60 testdata.536210 2024-08-19T15:52:01.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:01.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj61 testdata.536210 2024-08-19T15:52:01.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:01.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj62 testdata.536210 2024-08-19T15:52:02.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:02.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj63 testdata.536210 2024-08-19T15:52:02.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:02.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj64 testdata.536210 2024-08-19T15:52:02.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:02.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj65 testdata.536210 2024-08-19T15:52:02.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:02.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj66 testdata.536210 2024-08-19T15:52:02.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:02.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj67 testdata.536210 2024-08-19T15:52:02.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:02.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj68 testdata.536210 2024-08-19T15:52:02.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:02.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj69 testdata.536210 2024-08-19T15:52:03.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:03.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj70 testdata.536210 2024-08-19T15:52:03.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:03.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj71 testdata.536210 2024-08-19T15:52:03.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:03.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj72 testdata.536210 2024-08-19T15:52:03.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:03.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj73 testdata.536210 2024-08-19T15:52:03.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:03.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj74 testdata.536210 2024-08-19T15:52:03.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:03.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj75 testdata.536210 2024-08-19T15:52:03.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:03.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj76 testdata.536210 2024-08-19T15:52:04.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:04.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj77 testdata.536210 2024-08-19T15:52:04.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:04.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj78 testdata.536210 2024-08-19T15:52:04.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:04.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj79 testdata.536210 2024-08-19T15:52:04.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:04.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj80 testdata.536210 2024-08-19T15:52:04.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:04.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj81 testdata.536210 2024-08-19T15:52:04.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:04.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj82 testdata.536210 2024-08-19T15:52:04.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:04.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj83 testdata.536210 2024-08-19T15:52:04.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:04.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj84 testdata.536210 2024-08-19T15:52:05.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:05.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj85 testdata.536210 2024-08-19T15:52:05.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:05.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj86 testdata.536210 2024-08-19T15:52:05.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:05.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj87 testdata.536210 2024-08-19T15:52:05.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:05.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj88 testdata.536210 2024-08-19T15:52:05.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:05.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj89 testdata.536210 2024-08-19T15:52:05.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:05.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj90 testdata.536210 2024-08-19T15:52:06.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:06.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj91 testdata.536210 2024-08-19T15:52:06.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:06.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj92 testdata.536210 2024-08-19T15:52:06.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:06.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj93 testdata.536210 2024-08-19T15:52:06.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:06.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj94 testdata.536210 2024-08-19T15:52:06.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:06.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj95 testdata.536210 2024-08-19T15:52:06.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:06.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj96 testdata.536210 2024-08-19T15:52:06.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:06.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj97 testdata.536210 2024-08-19T15:52:07.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:07.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj98 testdata.536210 2024-08-19T15:52:07.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:07.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj99 testdata.536210 2024-08-19T15:52:07.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:07.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj100 testdata.536210 2024-08-19T15:52:07.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:07.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj101 testdata.536210 2024-08-19T15:52:07.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:07.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj102 testdata.536210 2024-08-19T15:52:07.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:07.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj103 testdata.536210 2024-08-19T15:52:07.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:07.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj104 testdata.536210 2024-08-19T15:52:08.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:08.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj105 testdata.536210 2024-08-19T15:52:08.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:08.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj106 testdata.536210 2024-08-19T15:52:08.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:08.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj107 testdata.536210 2024-08-19T15:52:08.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:08.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj108 testdata.536210 2024-08-19T15:52:08.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:08.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj109 testdata.536210 2024-08-19T15:52:08.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:08.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj110 testdata.536210 2024-08-19T15:52:08.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:08.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj111 testdata.536210 2024-08-19T15:52:09.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:09.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj112 testdata.536210 2024-08-19T15:52:09.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:09.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj113 testdata.536210 2024-08-19T15:52:09.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:09.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj114 testdata.536210 2024-08-19T15:52:09.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:09.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj115 testdata.536210 2024-08-19T15:52:09.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:09.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj116 testdata.536210 2024-08-19T15:52:09.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:09.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj117 testdata.536210 2024-08-19T15:52:09.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:09.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj118 testdata.536210 2024-08-19T15:52:09.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:09.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj119 testdata.536210 2024-08-19T15:52:10.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:10.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj120 testdata.536210 2024-08-19T15:52:10.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:10.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj121 testdata.536210 2024-08-19T15:52:10.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:10.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj122 testdata.536210 2024-08-19T15:52:10.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:10.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj123 testdata.536210 2024-08-19T15:52:10.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:10.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj124 testdata.536210 2024-08-19T15:52:10.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:10.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj125 testdata.536210 2024-08-19T15:52:10.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:10.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj126 testdata.536210 2024-08-19T15:52:11.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:11.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj127 testdata.536210 2024-08-19T15:52:11.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:11.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj128 testdata.536210 2024-08-19T15:52:11.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:11.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj129 testdata.536210 2024-08-19T15:52:11.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:11.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj130 testdata.536210 2024-08-19T15:52:11.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj131 testdata.536210 2024-08-19T15:52:11.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:11.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj132 testdata.536210 2024-08-19T15:52:11.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:11.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj133 testdata.536210 2024-08-19T15:52:12.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:12.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj134 testdata.536210 2024-08-19T15:52:12.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:12.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj135 testdata.536210 2024-08-19T15:52:12.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:12.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj136 testdata.536210 2024-08-19T15:52:12.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:12.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj137 testdata.536210 2024-08-19T15:52:12.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:12.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj138 testdata.536210 2024-08-19T15:52:12.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:12.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj139 testdata.536210 2024-08-19T15:52:12.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:12.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj140 testdata.536210 2024-08-19T15:52:13.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:13.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj141 testdata.536210 2024-08-19T15:52:13.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:13.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj142 testdata.536210 2024-08-19T15:52:13.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:13.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj143 testdata.536210 2024-08-19T15:52:13.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:13.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj144 testdata.536210 2024-08-19T15:52:13.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:13.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj145 testdata.536210 2024-08-19T15:52:13.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:13.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj146 testdata.536210 2024-08-19T15:52:13.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:13.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj147 testdata.536210 2024-08-19T15:52:14.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:14.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj148 testdata.536210 2024-08-19T15:52:14.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:14.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj149 testdata.536210 2024-08-19T15:52:14.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:14.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj150 testdata.536210 2024-08-19T15:52:14.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:14.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj151 testdata.536210 2024-08-19T15:52:14.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:14.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj152 testdata.536210 2024-08-19T15:52:14.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:14.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj153 testdata.536210 2024-08-19T15:52:14.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:14.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj154 testdata.536210 2024-08-19T15:52:15.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:15.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj155 testdata.536210 2024-08-19T15:52:15.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:15.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj156 testdata.536210 2024-08-19T15:52:15.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:15.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj157 testdata.536210 2024-08-19T15:52:15.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:15.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj158 testdata.536210 2024-08-19T15:52:15.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:15.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj159 testdata.536210 2024-08-19T15:52:15.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:15.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj160 testdata.536210 2024-08-19T15:52:16.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:16.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj161 testdata.536210 2024-08-19T15:52:16.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:16.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj162 testdata.536210 2024-08-19T15:52:16.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:16.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj163 testdata.536210 2024-08-19T15:52:16.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:16.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj164 testdata.536210 2024-08-19T15:52:16.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:16.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj165 testdata.536210 2024-08-19T15:52:16.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:16.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj166 testdata.536210 2024-08-19T15:52:16.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:16.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj167 testdata.536210 2024-08-19T15:52:17.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:17.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj168 testdata.536210 2024-08-19T15:52:17.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:17.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj169 testdata.536210 2024-08-19T15:52:17.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:17.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj170 testdata.536210 2024-08-19T15:52:17.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:17.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj171 testdata.536210 2024-08-19T15:52:17.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:17.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj172 testdata.536210 2024-08-19T15:52:17.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:17.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj173 testdata.536210 2024-08-19T15:52:17.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:17.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj174 testdata.536210 2024-08-19T15:52:18.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:18.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj175 testdata.536210 2024-08-19T15:52:18.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:18.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj176 testdata.536210 2024-08-19T15:52:18.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:18.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj177 testdata.536210 2024-08-19T15:52:18.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:18.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj178 testdata.536210 2024-08-19T15:52:18.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:18.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj179 testdata.536210 2024-08-19T15:52:18.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:18.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj180 testdata.536210 2024-08-19T15:52:18.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:18.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj181 testdata.536210 2024-08-19T15:52:18.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:18.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj182 testdata.536210 2024-08-19T15:52:19.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:19.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj183 testdata.536210 2024-08-19T15:52:19.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:19.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj184 testdata.536210 2024-08-19T15:52:19.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:19.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj185 testdata.536210 2024-08-19T15:52:19.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:19.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj186 testdata.536210 2024-08-19T15:52:19.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:19.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj187 testdata.536210 2024-08-19T15:52:19.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:19.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj188 testdata.536210 2024-08-19T15:52:19.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:19.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj189 testdata.536210 2024-08-19T15:52:20.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:20.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj190 testdata.536210 2024-08-19T15:52:20.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:20.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj191 testdata.536210 2024-08-19T15:52:20.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:20.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj192 testdata.536210 2024-08-19T15:52:20.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:20.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj193 testdata.536210 2024-08-19T15:52:20.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:20.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj194 testdata.536210 2024-08-19T15:52:20.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:20.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj195 testdata.536210 2024-08-19T15:52:20.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:20.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj196 testdata.536210 2024-08-19T15:52:21.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:21.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj197 testdata.536210 2024-08-19T15:52:21.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:21.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj198 testdata.536210 2024-08-19T15:52:21.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:21.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj199 testdata.536210 2024-08-19T15:52:21.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:21.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj200 testdata.536210 2024-08-19T15:52:21.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:21.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj201 testdata.536210 2024-08-19T15:52:21.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:21.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj202 testdata.536210 2024-08-19T15:52:22.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:22.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj203 testdata.536210 2024-08-19T15:52:22.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:22.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj204 testdata.536210 2024-08-19T15:52:22.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:22.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj205 testdata.536210 2024-08-19T15:52:22.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:22.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj206 testdata.536210 2024-08-19T15:52:22.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:22.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj207 testdata.536210 2024-08-19T15:52:22.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:22.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj208 testdata.536210 2024-08-19T15:52:22.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:22.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj209 testdata.536210 2024-08-19T15:52:22.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:22.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj210 testdata.536210 2024-08-19T15:52:23.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:23.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj211 testdata.536210 2024-08-19T15:52:23.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:23.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj212 testdata.536210 2024-08-19T15:52:23.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:23.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj213 testdata.536210 2024-08-19T15:52:23.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:23.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj214 testdata.536210 2024-08-19T15:52:23.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:23.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj215 testdata.536210 2024-08-19T15:52:23.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:23.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj216 testdata.536210 2024-08-19T15:52:23.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:23.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj217 testdata.536210 2024-08-19T15:52:24.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:24.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj218 testdata.536210 2024-08-19T15:52:24.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:24.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj219 testdata.536210 2024-08-19T15:52:24.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:24.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj220 testdata.536210 2024-08-19T15:52:24.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:24.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj221 testdata.536210 2024-08-19T15:52:24.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:24.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj222 testdata.536210 2024-08-19T15:52:24.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:24.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj223 testdata.536210 2024-08-19T15:52:24.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:24.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj224 testdata.536210 2024-08-19T15:52:25.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:25.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj225 testdata.536210 2024-08-19T15:52:25.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:25.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj226 testdata.536210 2024-08-19T15:52:25.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:25.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj227 testdata.536210 2024-08-19T15:52:25.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:25.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj228 testdata.536210 2024-08-19T15:52:25.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:25.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj229 testdata.536210 2024-08-19T15:52:25.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:25.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj230 testdata.536210 2024-08-19T15:52:25.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:25.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj231 testdata.536210 2024-08-19T15:52:26.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:26.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj232 testdata.536210 2024-08-19T15:52:26.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:26.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj233 testdata.536210 2024-08-19T15:52:26.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:26.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj234 testdata.536210 2024-08-19T15:52:26.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:26.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj235 testdata.536210 2024-08-19T15:52:26.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:26.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj236 testdata.536210 2024-08-19T15:52:26.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:26.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj237 testdata.536210 2024-08-19T15:52:26.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:26.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj238 testdata.536210 2024-08-19T15:52:27.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:27.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj239 testdata.536210 2024-08-19T15:52:27.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:27.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj240 testdata.536210 2024-08-19T15:52:27.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:27.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj241 testdata.536210 2024-08-19T15:52:27.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:27.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj242 testdata.536210 2024-08-19T15:52:27.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:27.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj243 testdata.536210 2024-08-19T15:52:27.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:27.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj244 testdata.536210 2024-08-19T15:52:27.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:27.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj245 testdata.536210 2024-08-19T15:52:28.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:28.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj246 testdata.536210 2024-08-19T15:52:28.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:28.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj247 testdata.536210 2024-08-19T15:52:28.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:28.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj248 testdata.536210 2024-08-19T15:52:28.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:28.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj249 testdata.536210 2024-08-19T15:52:28.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:28.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj250 testdata.536210 2024-08-19T15:52:28.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:28.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj251 testdata.536210 2024-08-19T15:52:28.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:28.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj252 testdata.536210 2024-08-19T15:52:28.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:28.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj253 testdata.536210 2024-08-19T15:52:29.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:29.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj254 testdata.536210 2024-08-19T15:52:29.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:29.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj255 testdata.536210 2024-08-19T15:52:29.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:29.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj256 testdata.536210 2024-08-19T15:52:29.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:29.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj257 testdata.536210 2024-08-19T15:52:29.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:29.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj258 testdata.536210 2024-08-19T15:52:29.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:29.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj259 testdata.536210 2024-08-19T15:52:30.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:30.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj260 testdata.536210 2024-08-19T15:52:30.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:30.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj261 testdata.536210 2024-08-19T15:52:30.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:30.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj262 testdata.536210 2024-08-19T15:52:30.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:30.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj263 testdata.536210 2024-08-19T15:52:30.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:30.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj264 testdata.536210 2024-08-19T15:52:30.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:30.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj265 testdata.536210 2024-08-19T15:52:30.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:30.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj266 testdata.536210 2024-08-19T15:52:30.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:30.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj267 testdata.536210 2024-08-19T15:52:31.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:31.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj268 testdata.536210 2024-08-19T15:52:31.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:31.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj269 testdata.536210 2024-08-19T15:52:31.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:31.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj270 testdata.536210 2024-08-19T15:52:31.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:31.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj271 testdata.536210 2024-08-19T15:52:31.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:31.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj272 testdata.536210 2024-08-19T15:52:31.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:31.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj273 testdata.536210 2024-08-19T15:52:32.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:32.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj274 testdata.536210 2024-08-19T15:52:32.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:32.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj275 testdata.536210 2024-08-19T15:52:32.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:32.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj276 testdata.536210 2024-08-19T15:52:32.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:32.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj277 testdata.536210 2024-08-19T15:52:32.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:32.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj278 testdata.536210 2024-08-19T15:52:32.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:32.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj279 testdata.536210 2024-08-19T15:52:32.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:32.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj280 testdata.536210 2024-08-19T15:52:33.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:33.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj281 testdata.536210 2024-08-19T15:52:33.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:33.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj282 testdata.536210 2024-08-19T15:52:33.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:33.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj283 testdata.536210 2024-08-19T15:52:33.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:33.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj284 testdata.536210 2024-08-19T15:52:33.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:33.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj285 testdata.536210 2024-08-19T15:52:33.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:33.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj286 testdata.536210 2024-08-19T15:52:33.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:33.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj287 testdata.536210 2024-08-19T15:52:33.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:33.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj288 testdata.536210 2024-08-19T15:52:34.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:34.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj289 testdata.536210 2024-08-19T15:52:34.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:34.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj290 testdata.536210 2024-08-19T15:52:34.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:34.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj291 testdata.536210 2024-08-19T15:52:34.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:34.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj292 testdata.536210 2024-08-19T15:52:34.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:34.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj293 testdata.536210 2024-08-19T15:52:34.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:34.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj294 testdata.536210 2024-08-19T15:52:34.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:34.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj295 testdata.536210 2024-08-19T15:52:35.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:35.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj296 testdata.536210 2024-08-19T15:52:35.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:35.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj297 testdata.536210 2024-08-19T15:52:35.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:35.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj298 testdata.536210 2024-08-19T15:52:35.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:35.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj299 testdata.536210 2024-08-19T15:52:35.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:35.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj300 testdata.536210 2024-08-19T15:52:35.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:35.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj301 testdata.536210 2024-08-19T15:52:35.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:35.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj302 testdata.536210 2024-08-19T15:52:36.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:36.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj303 testdata.536210 2024-08-19T15:52:36.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:36.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj304 testdata.536210 2024-08-19T15:52:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj305 testdata.536210 2024-08-19T15:52:36.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:36.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj306 testdata.536210 2024-08-19T15:52:36.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:36.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj307 testdata.536210 2024-08-19T15:52:36.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:36.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj308 testdata.536210 2024-08-19T15:52:36.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:36.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj309 testdata.536210 2024-08-19T15:52:37.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:37.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj310 testdata.536210 2024-08-19T15:52:37.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:37.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj311 testdata.536210 2024-08-19T15:52:37.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:37.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj312 testdata.536210 2024-08-19T15:52:37.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:37.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj313 testdata.536210 2024-08-19T15:52:37.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:37.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj314 testdata.536210 2024-08-19T15:52:37.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:37.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj315 testdata.536210 2024-08-19T15:52:37.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:37.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj316 testdata.536210 2024-08-19T15:52:38.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:38.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj317 testdata.536210 2024-08-19T15:52:38.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:38.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj318 testdata.536210 2024-08-19T15:52:38.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:38.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj319 testdata.536210 2024-08-19T15:52:38.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:38.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj320 testdata.536210 2024-08-19T15:52:38.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:38.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj321 testdata.536210 2024-08-19T15:52:38.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:38.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj322 testdata.536210 2024-08-19T15:52:38.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:38.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj323 testdata.536210 2024-08-19T15:52:39.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:39.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj324 testdata.536210 2024-08-19T15:52:39.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:39.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj325 testdata.536210 2024-08-19T15:52:39.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:39.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj326 testdata.536210 2024-08-19T15:52:39.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:39.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj327 testdata.536210 2024-08-19T15:52:39.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:39.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj328 testdata.536210 2024-08-19T15:52:39.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:39.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj329 testdata.536210 2024-08-19T15:52:39.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:39.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj330 testdata.536210 2024-08-19T15:52:40.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:40.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj331 testdata.536210 2024-08-19T15:52:40.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:40.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj332 testdata.536210 2024-08-19T15:52:40.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:40.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj333 testdata.536210 2024-08-19T15:52:40.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:40.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj334 testdata.536210 2024-08-19T15:52:40.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:40.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj335 testdata.536210 2024-08-19T15:52:40.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:40.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj336 testdata.536210 2024-08-19T15:52:40.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:40.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj337 testdata.536210 2024-08-19T15:52:41.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:41.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj338 testdata.536210 2024-08-19T15:52:41.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:41.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj339 testdata.536210 2024-08-19T15:52:41.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:41.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj340 testdata.536210 2024-08-19T15:52:41.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:41.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj341 testdata.536210 2024-08-19T15:52:41.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:41.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj342 testdata.536210 2024-08-19T15:52:41.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:41.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj343 testdata.536210 2024-08-19T15:52:41.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:41.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj344 testdata.536210 2024-08-19T15:52:41.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:41.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj345 testdata.536210 2024-08-19T15:52:42.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:42.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj346 testdata.536210 2024-08-19T15:52:42.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:42.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj347 testdata.536210 2024-08-19T15:52:42.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:42.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj348 testdata.536210 2024-08-19T15:52:42.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:42.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj349 testdata.536210 2024-08-19T15:52:42.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:42.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj350 testdata.536210 2024-08-19T15:52:42.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:42.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj351 testdata.536210 2024-08-19T15:52:42.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:42.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj352 testdata.536210 2024-08-19T15:52:43.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:43.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj353 testdata.536210 2024-08-19T15:52:43.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:43.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj354 testdata.536210 2024-08-19T15:52:43.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:43.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj355 testdata.536210 2024-08-19T15:52:43.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:43.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj356 testdata.536210 2024-08-19T15:52:43.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:43.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj357 testdata.536210 2024-08-19T15:52:43.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:43.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj358 testdata.536210 2024-08-19T15:52:43.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:43.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj359 testdata.536210 2024-08-19T15:52:44.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:44.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj360 testdata.536210 2024-08-19T15:52:44.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:44.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj361 testdata.536210 2024-08-19T15:52:44.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:44.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj362 testdata.536210 2024-08-19T15:52:44.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:44.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj363 testdata.536210 2024-08-19T15:52:44.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:44.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj364 testdata.536210 2024-08-19T15:52:44.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:44.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj365 testdata.536210 2024-08-19T15:52:44.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:44.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj366 testdata.536210 2024-08-19T15:52:45.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:45.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj367 testdata.536210 2024-08-19T15:52:45.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:45.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj368 testdata.536210 2024-08-19T15:52:45.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:45.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj369 testdata.536210 2024-08-19T15:52:45.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:45.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj370 testdata.536210 2024-08-19T15:52:45.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:45.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj371 testdata.536210 2024-08-19T15:52:45.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:45.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj372 testdata.536210 2024-08-19T15:52:45.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:45.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj373 testdata.536210 2024-08-19T15:52:46.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:46.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj374 testdata.536210 2024-08-19T15:52:46.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:46.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj375 testdata.536210 2024-08-19T15:52:46.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:46.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj376 testdata.536210 2024-08-19T15:52:46.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:46.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj377 testdata.536210 2024-08-19T15:52:46.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:46.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj378 testdata.536210 2024-08-19T15:52:46.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:46.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj379 testdata.536210 2024-08-19T15:52:46.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:46.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj380 testdata.536210 2024-08-19T15:52:47.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:47.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj381 testdata.536210 2024-08-19T15:52:47.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:47.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj382 testdata.536210 2024-08-19T15:52:47.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:47.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj383 testdata.536210 2024-08-19T15:52:47.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:47.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj384 testdata.536210 2024-08-19T15:52:47.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:47.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj385 testdata.536210 2024-08-19T15:52:47.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:47.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj386 testdata.536210 2024-08-19T15:52:48.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:48.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj387 testdata.536210 2024-08-19T15:52:48.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:48.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj388 testdata.536210 2024-08-19T15:52:48.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:48.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj389 testdata.536210 2024-08-19T15:52:48.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj390 testdata.536210 2024-08-19T15:52:48.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:48.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj391 testdata.536210 2024-08-19T15:52:48.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:48.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj392 testdata.536210 2024-08-19T15:52:48.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:48.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj393 testdata.536210 2024-08-19T15:52:49.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:49.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj394 testdata.536210 2024-08-19T15:52:49.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:49.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj395 testdata.536210 2024-08-19T15:52:49.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:49.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj396 testdata.536210 2024-08-19T15:52:49.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:49.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj397 testdata.536210 2024-08-19T15:52:49.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:49.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj398 testdata.536210 2024-08-19T15:52:49.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:49.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj399 testdata.536210 2024-08-19T15:52:49.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:49.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj400 testdata.536210 2024-08-19T15:52:50.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:50.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj401 testdata.536210 2024-08-19T15:52:50.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:50.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj402 testdata.536210 2024-08-19T15:52:50.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:50.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj403 testdata.536210 2024-08-19T15:52:50.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:50.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj404 testdata.536210 2024-08-19T15:52:50.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:50.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj405 testdata.536210 2024-08-19T15:52:50.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:50.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj406 testdata.536210 2024-08-19T15:52:50.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:50.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj407 testdata.536210 2024-08-19T15:52:50.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:50.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj408 testdata.536210 2024-08-19T15:52:51.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:51.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj409 testdata.536210 2024-08-19T15:52:51.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:51.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj410 testdata.536210 2024-08-19T15:52:51.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:51.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj411 testdata.536210 2024-08-19T15:52:51.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:51.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj412 testdata.536210 2024-08-19T15:52:51.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:51.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj413 testdata.536210 2024-08-19T15:52:51.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:51.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj414 testdata.536210 2024-08-19T15:52:51.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:51.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj415 testdata.536210 2024-08-19T15:52:52.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:52.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj416 testdata.536210 2024-08-19T15:52:52.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:52.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj417 testdata.536210 2024-08-19T15:52:52.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:52.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj418 testdata.536210 2024-08-19T15:52:52.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:52.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj419 testdata.536210 2024-08-19T15:52:52.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:52.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj420 testdata.536210 2024-08-19T15:52:52.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:52.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj421 testdata.536210 2024-08-19T15:52:53.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:53.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj422 testdata.536210 2024-08-19T15:52:53.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:53.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj423 testdata.536210 2024-08-19T15:52:53.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:53.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj424 testdata.536210 2024-08-19T15:52:53.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:53.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj425 testdata.536210 2024-08-19T15:52:53.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:53.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj426 testdata.536210 2024-08-19T15:52:53.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:53.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj427 testdata.536210 2024-08-19T15:52:53.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:53.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj428 testdata.536210 2024-08-19T15:52:54.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:54.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj429 testdata.536210 2024-08-19T15:52:54.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:54.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj430 testdata.536210 2024-08-19T15:52:54.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:54.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj431 testdata.536210 2024-08-19T15:52:54.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:54.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj432 testdata.536210 2024-08-19T15:52:54.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:54.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj433 testdata.536210 2024-08-19T15:52:54.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:54.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj434 testdata.536210 2024-08-19T15:52:54.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:54.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj435 testdata.536210 2024-08-19T15:52:54.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:54.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj436 testdata.536210 2024-08-19T15:52:55.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:55.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj437 testdata.536210 2024-08-19T15:52:55.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:55.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj438 testdata.536210 2024-08-19T15:52:55.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:55.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj439 testdata.536210 2024-08-19T15:52:55.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:55.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj440 testdata.536210 2024-08-19T15:52:55.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:55.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj441 testdata.536210 2024-08-19T15:52:55.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:55.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj442 testdata.536210 2024-08-19T15:52:55.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:55.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj443 testdata.536210 2024-08-19T15:52:56.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:56.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj444 testdata.536210 2024-08-19T15:52:56.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:56.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj445 testdata.536210 2024-08-19T15:52:56.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:56.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj446 testdata.536210 2024-08-19T15:52:56.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:56.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj447 testdata.536210 2024-08-19T15:52:56.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:56.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj448 testdata.536210 2024-08-19T15:52:56.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:56.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj449 testdata.536210 2024-08-19T15:52:56.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:56.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj450 testdata.536210 2024-08-19T15:52:57.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:57.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj451 testdata.536210 2024-08-19T15:52:57.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:57.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj452 testdata.536210 2024-08-19T15:52:57.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:57.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj453 testdata.536210 2024-08-19T15:52:57.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:57.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj454 testdata.536210 2024-08-19T15:52:57.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:57.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj455 testdata.536210 2024-08-19T15:52:57.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:57.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj456 testdata.536210 2024-08-19T15:52:58.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:58.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj457 testdata.536210 2024-08-19T15:52:58.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:58.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj458 testdata.536210 2024-08-19T15:52:58.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:58.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj459 testdata.536210 2024-08-19T15:52:58.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:58.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj460 testdata.536210 2024-08-19T15:52:58.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:58.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj461 testdata.536210 2024-08-19T15:52:58.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:58.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj462 testdata.536210 2024-08-19T15:52:58.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:58.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj463 testdata.536210 2024-08-19T15:52:58.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:58.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj464 testdata.536210 2024-08-19T15:52:59.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:59.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj465 testdata.536210 2024-08-19T15:52:59.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:59.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj466 testdata.536210 2024-08-19T15:52:59.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:59.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj467 testdata.536210 2024-08-19T15:52:59.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:59.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj468 testdata.536210 2024-08-19T15:52:59.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:59.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj469 testdata.536210 2024-08-19T15:52:59.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:52:59.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj470 testdata.536210 2024-08-19T15:53:00.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:00.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj471 testdata.536210 2024-08-19T15:53:00.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:00.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj472 testdata.536210 2024-08-19T15:53:00.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:00.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj473 testdata.536210 2024-08-19T15:53:00.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:00.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj474 testdata.536210 2024-08-19T15:53:00.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:00.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj475 testdata.536210 2024-08-19T15:53:00.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:00.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj476 testdata.536210 2024-08-19T15:53:00.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:00.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj477 testdata.536210 2024-08-19T15:53:01.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:01.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj478 testdata.536210 2024-08-19T15:53:01.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:01.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj479 testdata.536210 2024-08-19T15:53:01.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:01.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj480 testdata.536210 2024-08-19T15:53:01.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:01.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj481 testdata.536210 2024-08-19T15:53:01.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:01.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj482 testdata.536210 2024-08-19T15:53:01.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:01.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj483 testdata.536210 2024-08-19T15:53:01.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:01.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj484 testdata.536210 2024-08-19T15:53:01.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:01.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj485 testdata.536210 2024-08-19T15:53:02.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:02.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj486 testdata.536210 2024-08-19T15:53:02.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:02.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj487 testdata.536210 2024-08-19T15:53:02.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:02.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj488 testdata.536210 2024-08-19T15:53:02.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:02.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj489 testdata.536210 2024-08-19T15:53:02.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:02.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj490 testdata.536210 2024-08-19T15:53:02.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:02.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj491 testdata.536210 2024-08-19T15:53:03.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:03.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj492 testdata.536210 2024-08-19T15:53:03.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:03.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj493 testdata.536210 2024-08-19T15:53:03.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:03.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj494 testdata.536210 2024-08-19T15:53:03.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:03.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj495 testdata.536210 2024-08-19T15:53:03.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:03.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj496 testdata.536210 2024-08-19T15:53:03.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:03.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj497 testdata.536210 2024-08-19T15:53:03.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:03.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj498 testdata.536210 2024-08-19T15:53:04.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:04.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj499 testdata.536210 2024-08-19T15:53:04.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:04.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj500 testdata.536210 2024-08-19T15:53:04.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:04.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj501 testdata.536210 2024-08-19T15:53:04.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:04.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj502 testdata.536210 2024-08-19T15:53:04.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:04.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj503 testdata.536210 2024-08-19T15:53:04.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:04.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj504 testdata.536210 2024-08-19T15:53:04.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:04.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj505 testdata.536210 2024-08-19T15:53:04.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:04.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj506 testdata.536210 2024-08-19T15:53:05.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:05.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj507 testdata.536210 2024-08-19T15:53:05.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:05.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj508 testdata.536210 2024-08-19T15:53:05.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:05.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj509 testdata.536210 2024-08-19T15:53:05.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:05.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj510 testdata.536210 2024-08-19T15:53:05.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:05.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj511 testdata.536210 2024-08-19T15:53:05.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:05.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj512 testdata.536210 2024-08-19T15:53:05.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:05.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj513 testdata.536210 2024-08-19T15:53:06.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:06.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj514 testdata.536210 2024-08-19T15:53:06.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:06.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj515 testdata.536210 2024-08-19T15:53:06.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:06.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj516 testdata.536210 2024-08-19T15:53:06.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:06.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj517 testdata.536210 2024-08-19T15:53:06.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:06.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj518 testdata.536210 2024-08-19T15:53:06.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:06.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj519 testdata.536210 2024-08-19T15:53:06.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:06.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj520 testdata.536210 2024-08-19T15:53:07.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:07.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj521 testdata.536210 2024-08-19T15:53:07.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:07.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj522 testdata.536210 2024-08-19T15:53:07.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:07.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj523 testdata.536210 2024-08-19T15:53:07.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:07.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj524 testdata.536210 2024-08-19T15:53:07.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:07.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj525 testdata.536210 2024-08-19T15:53:07.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:07.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj526 testdata.536210 2024-08-19T15:53:07.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:07.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj527 testdata.536210 2024-08-19T15:53:08.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:08.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj528 testdata.536210 2024-08-19T15:53:08.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:08.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj529 testdata.536210 2024-08-19T15:53:08.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:08.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj530 testdata.536210 2024-08-19T15:53:08.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:08.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj531 testdata.536210 2024-08-19T15:53:08.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:08.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj532 testdata.536210 2024-08-19T15:53:08.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:08.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj533 testdata.536210 2024-08-19T15:53:08.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:08.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj534 testdata.536210 2024-08-19T15:53:09.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:09.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj535 testdata.536210 2024-08-19T15:53:09.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:09.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj536 testdata.536210 2024-08-19T15:53:09.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:09.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj537 testdata.536210 2024-08-19T15:53:09.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:09.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj538 testdata.536210 2024-08-19T15:53:09.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:09.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj539 testdata.536210 2024-08-19T15:53:09.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:09.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj540 testdata.536210 2024-08-19T15:53:09.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:09.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj541 testdata.536210 2024-08-19T15:53:10.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:10.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj542 testdata.536210 2024-08-19T15:53:10.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:10.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj543 testdata.536210 2024-08-19T15:53:10.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:10.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj544 testdata.536210 2024-08-19T15:53:10.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:10.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj545 testdata.536210 2024-08-19T15:53:10.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:10.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj546 testdata.536210 2024-08-19T15:53:10.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:10.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj547 testdata.536210 2024-08-19T15:53:10.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:10.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj548 testdata.536210 2024-08-19T15:53:11.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:11.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj549 testdata.536210 2024-08-19T15:53:11.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:11.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj550 testdata.536210 2024-08-19T15:53:11.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:11.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj551 testdata.536210 2024-08-19T15:53:11.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:11.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj552 testdata.536210 2024-08-19T15:53:11.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:11.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj553 testdata.536210 2024-08-19T15:53:11.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:11.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj554 testdata.536210 2024-08-19T15:53:11.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:11.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj555 testdata.536210 2024-08-19T15:53:11.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:11.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj556 testdata.536210 2024-08-19T15:53:12.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:12.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj557 testdata.536210 2024-08-19T15:53:12.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:12.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj558 testdata.536210 2024-08-19T15:53:12.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:12.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj559 testdata.536210 2024-08-19T15:53:12.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:12.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj560 testdata.536210 2024-08-19T15:53:12.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:12.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj561 testdata.536210 2024-08-19T15:53:12.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:12.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj562 testdata.536210 2024-08-19T15:53:12.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:12.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj563 testdata.536210 2024-08-19T15:53:13.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:13.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj564 testdata.536210 2024-08-19T15:53:13.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:13.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj565 testdata.536210 2024-08-19T15:53:13.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:13.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj566 testdata.536210 2024-08-19T15:53:13.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:13.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj567 testdata.536210 2024-08-19T15:53:13.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:13.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj568 testdata.536210 2024-08-19T15:53:13.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:13.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj569 testdata.536210 2024-08-19T15:53:13.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:13.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj570 testdata.536210 2024-08-19T15:53:13.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:13.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj571 testdata.536210 2024-08-19T15:53:14.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:14.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj572 testdata.536210 2024-08-19T15:53:14.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:14.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj573 testdata.536210 2024-08-19T15:53:14.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:14.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj574 testdata.536210 2024-08-19T15:53:14.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:14.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj575 testdata.536210 2024-08-19T15:53:14.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:14.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj576 testdata.536210 2024-08-19T15:53:14.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:14.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj577 testdata.536210 2024-08-19T15:53:14.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:14.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj578 testdata.536210 2024-08-19T15:53:15.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:15.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj579 testdata.536210 2024-08-19T15:53:15.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:15.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj580 testdata.536210 2024-08-19T15:53:15.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:15.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj581 testdata.536210 2024-08-19T15:53:15.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:15.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj582 testdata.536210 2024-08-19T15:53:15.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:15.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj583 testdata.536210 2024-08-19T15:53:15.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:15.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj584 testdata.536210 2024-08-19T15:53:15.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:15.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj585 testdata.536210 2024-08-19T15:53:16.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:16.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj586 testdata.536210 2024-08-19T15:53:16.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:16.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj587 testdata.536210 2024-08-19T15:53:16.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:16.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj588 testdata.536210 2024-08-19T15:53:16.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:16.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj589 testdata.536210 2024-08-19T15:53:16.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:16.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj590 testdata.536210 2024-08-19T15:53:16.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:16.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj591 testdata.536210 2024-08-19T15:53:16.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:16.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj592 testdata.536210 2024-08-19T15:53:17.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:17.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj593 testdata.536210 2024-08-19T15:53:17.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:17.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj594 testdata.536210 2024-08-19T15:53:17.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:17.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj595 testdata.536210 2024-08-19T15:53:17.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:17.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj596 testdata.536210 2024-08-19T15:53:17.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:17.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj597 testdata.536210 2024-08-19T15:53:17.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:17.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj598 testdata.536210 2024-08-19T15:53:17.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:17.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj599 testdata.536210 2024-08-19T15:53:18.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:18.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj600 testdata.536210 2024-08-19T15:53:18.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:18.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj601 testdata.536210 2024-08-19T15:53:18.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:18.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj602 testdata.536210 2024-08-19T15:53:18.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:18.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj603 testdata.536210 2024-08-19T15:53:18.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:18.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj604 testdata.536210 2024-08-19T15:53:18.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:18.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj605 testdata.536210 2024-08-19T15:53:19.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:19.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj606 testdata.536210 2024-08-19T15:53:19.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:19.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj607 testdata.536210 2024-08-19T15:53:19.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:19.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj608 testdata.536210 2024-08-19T15:53:19.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:19.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj609 testdata.536210 2024-08-19T15:53:19.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:19.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj610 testdata.536210 2024-08-19T15:53:19.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:19.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj611 testdata.536210 2024-08-19T15:53:19.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:19.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj612 testdata.536210 2024-08-19T15:53:20.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:20.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj613 testdata.536210 2024-08-19T15:53:20.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:20.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj614 testdata.536210 2024-08-19T15:53:20.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:20.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj615 testdata.536210 2024-08-19T15:53:20.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:20.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj616 testdata.536210 2024-08-19T15:53:20.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:20.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj617 testdata.536210 2024-08-19T15:53:20.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:20.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj618 testdata.536210 2024-08-19T15:53:20.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:20.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj619 testdata.536210 2024-08-19T15:53:20.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:20.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj620 testdata.536210 2024-08-19T15:53:21.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:21.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj621 testdata.536210 2024-08-19T15:53:21.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:21.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj622 testdata.536210 2024-08-19T15:53:21.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:21.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj623 testdata.536210 2024-08-19T15:53:21.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:21.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj624 testdata.536210 2024-08-19T15:53:21.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:21.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj625 testdata.536210 2024-08-19T15:53:21.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:21.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj626 testdata.536210 2024-08-19T15:53:21.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:21.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj627 testdata.536210 2024-08-19T15:53:22.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:22.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj628 testdata.536210 2024-08-19T15:53:22.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:22.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj629 testdata.536210 2024-08-19T15:53:22.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:22.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj630 testdata.536210 2024-08-19T15:53:22.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:22.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj631 testdata.536210 2024-08-19T15:53:22.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:22.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj632 testdata.536210 2024-08-19T15:53:22.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:22.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj633 testdata.536210 2024-08-19T15:53:22.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:22.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj634 testdata.536210 2024-08-19T15:53:23.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:23.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj635 testdata.536210 2024-08-19T15:53:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj636 testdata.536210 2024-08-19T15:53:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj637 testdata.536210 2024-08-19T15:53:23.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:23.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj638 testdata.536210 2024-08-19T15:53:23.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:23.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj639 testdata.536210 2024-08-19T15:53:23.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:23.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj640 testdata.536210 2024-08-19T15:53:23.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:23.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj641 testdata.536210 2024-08-19T15:53:24.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:24.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj642 testdata.536210 2024-08-19T15:53:24.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:24.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj643 testdata.536210 2024-08-19T15:53:24.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:24.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj644 testdata.536210 2024-08-19T15:53:24.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:24.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj645 testdata.536210 2024-08-19T15:53:24.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:24.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj646 testdata.536210 2024-08-19T15:53:24.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:24.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj647 testdata.536210 2024-08-19T15:53:24.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:24.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj648 testdata.536210 2024-08-19T15:53:25.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:25.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj649 testdata.536210 2024-08-19T15:53:25.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:25.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj650 testdata.536210 2024-08-19T15:53:25.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:25.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj651 testdata.536210 2024-08-19T15:53:25.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:25.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj652 testdata.536210 2024-08-19T15:53:25.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:25.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj653 testdata.536210 2024-08-19T15:53:25.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:25.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj654 testdata.536210 2024-08-19T15:53:25.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:25.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj655 testdata.536210 2024-08-19T15:53:26.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:26.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj656 testdata.536210 2024-08-19T15:53:26.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:26.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj657 testdata.536210 2024-08-19T15:53:26.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:26.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj658 testdata.536210 2024-08-19T15:53:26.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:26.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj659 testdata.536210 2024-08-19T15:53:26.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:26.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj660 testdata.536210 2024-08-19T15:53:26.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:26.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj661 testdata.536210 2024-08-19T15:53:26.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:26.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj662 testdata.536210 2024-08-19T15:53:27.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:27.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj663 testdata.536210 2024-08-19T15:53:27.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:27.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj664 testdata.536210 2024-08-19T15:53:27.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:27.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj665 testdata.536210 2024-08-19T15:53:27.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:27.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj666 testdata.536210 2024-08-19T15:53:27.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:27.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj667 testdata.536210 2024-08-19T15:53:27.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:27.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj668 testdata.536210 2024-08-19T15:53:27.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:27.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj669 testdata.536210 2024-08-19T15:53:27.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:27.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj670 testdata.536210 2024-08-19T15:53:28.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:28.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj671 testdata.536210 2024-08-19T15:53:28.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:28.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj672 testdata.536210 2024-08-19T15:53:28.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:28.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj673 testdata.536210 2024-08-19T15:53:28.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:28.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj674 testdata.536210 2024-08-19T15:53:28.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:28.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj675 testdata.536210 2024-08-19T15:53:28.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:28.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj676 testdata.536210 2024-08-19T15:53:28.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:28.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj677 testdata.536210 2024-08-19T15:53:29.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:29.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj678 testdata.536210 2024-08-19T15:53:29.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:29.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj679 testdata.536210 2024-08-19T15:53:29.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:29.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj680 testdata.536210 2024-08-19T15:53:29.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:29.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj681 testdata.536210 2024-08-19T15:53:29.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:29.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj682 testdata.536210 2024-08-19T15:53:29.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:29.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj683 testdata.536210 2024-08-19T15:53:29.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:29.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj684 testdata.536210 2024-08-19T15:53:29.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:29.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj685 testdata.536210 2024-08-19T15:53:30.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:30.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj686 testdata.536210 2024-08-19T15:53:30.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:30.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj687 testdata.536210 2024-08-19T15:53:30.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:30.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj688 testdata.536210 2024-08-19T15:53:30.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:30.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj689 testdata.536210 2024-08-19T15:53:30.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:30.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj690 testdata.536210 2024-08-19T15:53:30.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:30.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj691 testdata.536210 2024-08-19T15:53:30.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:30.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj692 testdata.536210 2024-08-19T15:53:30.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:30.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj693 testdata.536210 2024-08-19T15:53:31.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:31.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj694 testdata.536210 2024-08-19T15:53:31.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:31.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj695 testdata.536210 2024-08-19T15:53:31.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:31.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj696 testdata.536210 2024-08-19T15:53:31.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:31.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj697 testdata.536210 2024-08-19T15:53:31.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:31.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj698 testdata.536210 2024-08-19T15:53:31.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:31.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj699 testdata.536210 2024-08-19T15:53:31.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:31.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj700 testdata.536210 2024-08-19T15:53:32.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:32.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj701 testdata.536210 2024-08-19T15:53:32.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:32.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj702 testdata.536210 2024-08-19T15:53:32.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:32.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj703 testdata.536210 2024-08-19T15:53:32.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:32.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj704 testdata.536210 2024-08-19T15:53:32.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:32.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj705 testdata.536210 2024-08-19T15:53:32.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:32.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj706 testdata.536210 2024-08-19T15:53:32.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:32.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj707 testdata.536210 2024-08-19T15:53:32.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:32.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj708 testdata.536210 2024-08-19T15:53:33.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:33.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj709 testdata.536210 2024-08-19T15:53:33.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:33.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj710 testdata.536210 2024-08-19T15:53:33.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:33.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj711 testdata.536210 2024-08-19T15:53:33.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:33.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj712 testdata.536210 2024-08-19T15:53:33.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:33.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj713 testdata.536210 2024-08-19T15:53:33.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:33.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj714 testdata.536210 2024-08-19T15:53:33.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:33.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj715 testdata.536210 2024-08-19T15:53:34.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:34.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj716 testdata.536210 2024-08-19T15:53:34.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:34.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj717 testdata.536210 2024-08-19T15:53:34.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:34.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj718 testdata.536210 2024-08-19T15:53:34.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:34.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj719 testdata.536210 2024-08-19T15:53:34.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:34.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj720 testdata.536210 2024-08-19T15:53:34.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:34.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj721 testdata.536210 2024-08-19T15:53:34.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:34.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj722 testdata.536210 2024-08-19T15:53:34.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:34.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj723 testdata.536210 2024-08-19T15:53:35.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:35.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj724 testdata.536210 2024-08-19T15:53:35.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:35.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj725 testdata.536210 2024-08-19T15:53:35.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:35.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj726 testdata.536210 2024-08-19T15:53:35.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:35.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj727 testdata.536210 2024-08-19T15:53:35.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:35.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj728 testdata.536210 2024-08-19T15:53:35.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:35.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj729 testdata.536210 2024-08-19T15:53:35.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:35.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj730 testdata.536210 2024-08-19T15:53:36.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:36.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj731 testdata.536210 2024-08-19T15:53:36.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:36.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj732 testdata.536210 2024-08-19T15:53:36.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:36.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj733 testdata.536210 2024-08-19T15:53:36.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:36.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj734 testdata.536210 2024-08-19T15:53:36.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:36.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj735 testdata.536210 2024-08-19T15:53:36.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:36.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj736 testdata.536210 2024-08-19T15:53:36.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:36.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj737 testdata.536210 2024-08-19T15:53:36.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:36.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj738 testdata.536210 2024-08-19T15:53:37.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:37.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj739 testdata.536210 2024-08-19T15:53:37.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:37.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj740 testdata.536210 2024-08-19T15:53:37.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:37.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj741 testdata.536210 2024-08-19T15:53:37.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:37.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj742 testdata.536210 2024-08-19T15:53:37.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:37.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj743 testdata.536210 2024-08-19T15:53:37.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:37.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj744 testdata.536210 2024-08-19T15:53:37.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:37.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj745 testdata.536210 2024-08-19T15:53:38.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:38.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj746 testdata.536210 2024-08-19T15:53:38.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:38.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj747 testdata.536210 2024-08-19T15:53:38.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:38.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj748 testdata.536210 2024-08-19T15:53:38.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:38.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj749 testdata.536210 2024-08-19T15:53:38.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:38.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj750 testdata.536210 2024-08-19T15:53:38.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:38.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj751 testdata.536210 2024-08-19T15:53:38.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:38.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj752 testdata.536210 2024-08-19T15:53:39.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:39.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj753 testdata.536210 2024-08-19T15:53:39.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:39.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj754 testdata.536210 2024-08-19T15:53:39.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:39.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj755 testdata.536210 2024-08-19T15:53:39.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:39.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj756 testdata.536210 2024-08-19T15:53:39.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:39.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj757 testdata.536210 2024-08-19T15:53:39.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:39.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj758 testdata.536210 2024-08-19T15:53:39.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:39.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj759 testdata.536210 2024-08-19T15:53:39.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:39.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj760 testdata.536210 2024-08-19T15:53:40.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:40.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj761 testdata.536210 2024-08-19T15:53:40.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:40.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj762 testdata.536210 2024-08-19T15:53:40.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:40.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj763 testdata.536210 2024-08-19T15:53:40.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:40.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj764 testdata.536210 2024-08-19T15:53:40.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:40.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj765 testdata.536210 2024-08-19T15:53:40.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:40.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj766 testdata.536210 2024-08-19T15:53:40.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:40.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj767 testdata.536210 2024-08-19T15:53:41.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:41.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj768 testdata.536210 2024-08-19T15:53:41.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:41.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj769 testdata.536210 2024-08-19T15:53:41.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:41.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj770 testdata.536210 2024-08-19T15:53:41.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:41.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj771 testdata.536210 2024-08-19T15:53:41.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:41.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj772 testdata.536210 2024-08-19T15:53:41.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:41.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj773 testdata.536210 2024-08-19T15:53:41.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:41.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj774 testdata.536210 2024-08-19T15:53:41.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:41.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj775 testdata.536210 2024-08-19T15:53:42.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:42.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj776 testdata.536210 2024-08-19T15:53:42.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:42.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj777 testdata.536210 2024-08-19T15:53:42.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:42.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj778 testdata.536210 2024-08-19T15:53:42.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:42.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj779 testdata.536210 2024-08-19T15:53:42.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:42.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj780 testdata.536210 2024-08-19T15:53:42.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:42.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj781 testdata.536210 2024-08-19T15:53:42.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:42.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj782 testdata.536210 2024-08-19T15:53:42.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:42.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj783 testdata.536210 2024-08-19T15:53:43.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:43.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj784 testdata.536210 2024-08-19T15:53:43.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:43.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj785 testdata.536210 2024-08-19T15:53:43.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:43.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj786 testdata.536210 2024-08-19T15:53:43.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:43.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj787 testdata.536210 2024-08-19T15:53:43.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:43.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj788 testdata.536210 2024-08-19T15:53:43.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:43.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj789 testdata.536210 2024-08-19T15:53:43.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:43.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj790 testdata.536210 2024-08-19T15:53:44.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:44.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj791 testdata.536210 2024-08-19T15:53:44.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:44.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj792 testdata.536210 2024-08-19T15:53:44.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:44.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj793 testdata.536210 2024-08-19T15:53:44.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:44.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj794 testdata.536210 2024-08-19T15:53:44.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:44.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj795 testdata.536210 2024-08-19T15:53:44.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:44.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj796 testdata.536210 2024-08-19T15:53:44.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:44.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj797 testdata.536210 2024-08-19T15:53:44.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:44.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj798 testdata.536210 2024-08-19T15:53:45.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:45.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj799 testdata.536210 2024-08-19T15:53:45.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:45.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj800 testdata.536210 2024-08-19T15:53:45.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:45.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj801 testdata.536210 2024-08-19T15:53:45.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:45.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj802 testdata.536210 2024-08-19T15:53:45.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:45.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj803 testdata.536210 2024-08-19T15:53:45.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:45.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj804 testdata.536210 2024-08-19T15:53:45.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:45.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj805 testdata.536210 2024-08-19T15:53:46.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:46.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj806 testdata.536210 2024-08-19T15:53:46.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:46.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj807 testdata.536210 2024-08-19T15:53:46.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:46.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj808 testdata.536210 2024-08-19T15:53:46.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:46.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj809 testdata.536210 2024-08-19T15:53:46.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:46.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj810 testdata.536210 2024-08-19T15:53:46.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:46.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj811 testdata.536210 2024-08-19T15:53:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj812 testdata.536210 2024-08-19T15:53:46.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:46.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj813 testdata.536210 2024-08-19T15:53:47.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:47.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj814 testdata.536210 2024-08-19T15:53:47.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:47.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj815 testdata.536210 2024-08-19T15:53:47.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:47.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj816 testdata.536210 2024-08-19T15:53:47.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:47.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj817 testdata.536210 2024-08-19T15:53:47.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:47.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj818 testdata.536210 2024-08-19T15:53:47.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:47.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj819 testdata.536210 2024-08-19T15:53:47.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:47.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj820 testdata.536210 2024-08-19T15:53:48.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:48.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj821 testdata.536210 2024-08-19T15:53:48.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:48.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj822 testdata.536210 2024-08-19T15:53:48.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:48.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj823 testdata.536210 2024-08-19T15:53:48.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:48.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj824 testdata.536210 2024-08-19T15:53:48.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:48.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj825 testdata.536210 2024-08-19T15:53:48.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:48.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj826 testdata.536210 2024-08-19T15:53:48.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:48.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj827 testdata.536210 2024-08-19T15:53:48.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:48.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj828 testdata.536210 2024-08-19T15:53:49.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:49.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj829 testdata.536210 2024-08-19T15:53:49.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:49.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj830 testdata.536210 2024-08-19T15:53:49.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:49.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj831 testdata.536210 2024-08-19T15:53:49.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:49.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj832 testdata.536210 2024-08-19T15:53:49.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:49.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj833 testdata.536210 2024-08-19T15:53:50.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:50.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj834 testdata.536210 2024-08-19T15:53:50.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:50.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj835 testdata.536210 2024-08-19T15:53:50.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:50.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj836 testdata.536210 2024-08-19T15:53:50.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:50.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj837 testdata.536210 2024-08-19T15:53:50.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:50.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj838 testdata.536210 2024-08-19T15:53:50.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:50.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj839 testdata.536210 2024-08-19T15:53:50.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:50.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj840 testdata.536210 2024-08-19T15:53:50.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:50.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj841 testdata.536210 2024-08-19T15:53:51.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:51.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj842 testdata.536210 2024-08-19T15:53:51.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:51.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj843 testdata.536210 2024-08-19T15:53:51.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:51.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj844 testdata.536210 2024-08-19T15:53:51.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:51.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj845 testdata.536210 2024-08-19T15:53:51.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:51.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj846 testdata.536210 2024-08-19T15:53:51.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:51.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj847 testdata.536210 2024-08-19T15:53:51.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:51.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj848 testdata.536210 2024-08-19T15:53:52.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:52.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj849 testdata.536210 2024-08-19T15:53:52.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:52.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj850 testdata.536210 2024-08-19T15:53:52.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:52.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj851 testdata.536210 2024-08-19T15:53:52.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:52.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj852 testdata.536210 2024-08-19T15:53:52.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:52.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj853 testdata.536210 2024-08-19T15:53:52.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:52.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj854 testdata.536210 2024-08-19T15:53:52.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:52.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj855 testdata.536210 2024-08-19T15:53:53.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:53.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj856 testdata.536210 2024-08-19T15:53:53.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:53.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj857 testdata.536210 2024-08-19T15:53:53.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:53.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj858 testdata.536210 2024-08-19T15:53:53.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:53.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj859 testdata.536210 2024-08-19T15:53:53.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:53.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj860 testdata.536210 2024-08-19T15:53:53.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:53.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj861 testdata.536210 2024-08-19T15:53:53.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:53.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj862 testdata.536210 2024-08-19T15:53:53.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:53.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj863 testdata.536210 2024-08-19T15:53:54.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:54.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj864 testdata.536210 2024-08-19T15:53:54.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:54.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj865 testdata.536210 2024-08-19T15:53:54.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:54.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj866 testdata.536210 2024-08-19T15:53:54.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:54.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj867 testdata.536210 2024-08-19T15:53:54.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:54.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj868 testdata.536210 2024-08-19T15:53:54.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:54.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj869 testdata.536210 2024-08-19T15:53:54.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:54.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj870 testdata.536210 2024-08-19T15:53:55.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:55.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj871 testdata.536210 2024-08-19T15:53:55.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:55.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj872 testdata.536210 2024-08-19T15:53:55.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:55.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj873 testdata.536210 2024-08-19T15:53:55.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:55.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj874 testdata.536210 2024-08-19T15:53:55.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:55.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj875 testdata.536210 2024-08-19T15:53:55.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:55.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj876 testdata.536210 2024-08-19T15:53:56.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:56.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj877 testdata.536210 2024-08-19T15:53:56.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:56.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj878 testdata.536210 2024-08-19T15:53:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj879 testdata.536210 2024-08-19T15:53:56.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:56.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj880 testdata.536210 2024-08-19T15:53:56.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:56.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj881 testdata.536210 2024-08-19T15:53:56.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:56.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj882 testdata.536210 2024-08-19T15:53:56.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:56.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj883 testdata.536210 2024-08-19T15:53:57.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:57.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj884 testdata.536210 2024-08-19T15:53:57.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:57.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj885 testdata.536210 2024-08-19T15:53:57.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:57.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj886 testdata.536210 2024-08-19T15:53:57.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:57.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj887 testdata.536210 2024-08-19T15:53:57.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:57.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj888 testdata.536210 2024-08-19T15:53:57.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:57.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj889 testdata.536210 2024-08-19T15:53:57.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:57.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj890 testdata.536210 2024-08-19T15:53:58.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:58.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj891 testdata.536210 2024-08-19T15:53:58.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:58.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj892 testdata.536210 2024-08-19T15:53:58.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:58.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj893 testdata.536210 2024-08-19T15:53:58.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:58.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj894 testdata.536210 2024-08-19T15:53:58.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:58.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj895 testdata.536210 2024-08-19T15:53:58.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:58.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj896 testdata.536210 2024-08-19T15:53:58.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:58.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj897 testdata.536210 2024-08-19T15:53:58.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:58.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj898 testdata.536210 2024-08-19T15:53:59.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:59.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj899 testdata.536210 2024-08-19T15:53:59.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:59.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj900 testdata.536210 2024-08-19T15:53:59.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:59.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj901 testdata.536210 2024-08-19T15:53:59.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:59.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj902 testdata.536210 2024-08-19T15:53:59.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:59.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj903 testdata.536210 2024-08-19T15:53:59.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:59.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj904 testdata.536210 2024-08-19T15:53:59.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:53:59.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj905 testdata.536210 2024-08-19T15:54:00.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:00.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj906 testdata.536210 2024-08-19T15:54:00.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:00.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj907 testdata.536210 2024-08-19T15:54:00.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:00.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj908 testdata.536210 2024-08-19T15:54:00.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:00.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj909 testdata.536210 2024-08-19T15:54:00.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:00.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj910 testdata.536210 2024-08-19T15:54:00.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:00.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj911 testdata.536210 2024-08-19T15:54:00.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:00.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj912 testdata.536210 2024-08-19T15:54:01.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:01.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj913 testdata.536210 2024-08-19T15:54:01.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:01.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj914 testdata.536210 2024-08-19T15:54:01.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:01.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj915 testdata.536210 2024-08-19T15:54:01.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:01.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj916 testdata.536210 2024-08-19T15:54:01.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:01.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj917 testdata.536210 2024-08-19T15:54:01.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:01.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj918 testdata.536210 2024-08-19T15:54:01.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:01.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj919 testdata.536210 2024-08-19T15:54:02.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:02.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj920 testdata.536210 2024-08-19T15:54:02.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:02.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj921 testdata.536210 2024-08-19T15:54:02.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:02.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj922 testdata.536210 2024-08-19T15:54:02.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:02.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj923 testdata.536210 2024-08-19T15:54:02.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:02.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj924 testdata.536210 2024-08-19T15:54:02.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:02.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj925 testdata.536210 2024-08-19T15:54:02.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:02.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj926 testdata.536210 2024-08-19T15:54:03.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:03.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj927 testdata.536210 2024-08-19T15:54:03.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:03.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj928 testdata.536210 2024-08-19T15:54:03.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:03.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj929 testdata.536210 2024-08-19T15:54:03.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:03.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj930 testdata.536210 2024-08-19T15:54:03.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:03.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj931 testdata.536210 2024-08-19T15:54:03.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:03.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj932 testdata.536210 2024-08-19T15:54:03.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:03.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj933 testdata.536210 2024-08-19T15:54:03.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:03.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj934 testdata.536210 2024-08-19T15:54:04.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:04.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj935 testdata.536210 2024-08-19T15:54:04.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:04.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj936 testdata.536210 2024-08-19T15:54:04.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:04.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj937 testdata.536210 2024-08-19T15:54:04.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:04.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj938 testdata.536210 2024-08-19T15:54:04.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:04.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj939 testdata.536210 2024-08-19T15:54:04.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:04.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj940 testdata.536210 2024-08-19T15:54:04.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:04.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj941 testdata.536210 2024-08-19T15:54:05.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:05.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj942 testdata.536210 2024-08-19T15:54:05.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:05.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj943 testdata.536210 2024-08-19T15:54:05.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:05.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj944 testdata.536210 2024-08-19T15:54:05.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:05.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj945 testdata.536210 2024-08-19T15:54:05.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:05.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj946 testdata.536210 2024-08-19T15:54:05.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:05.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj947 testdata.536210 2024-08-19T15:54:05.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:05.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj948 testdata.536210 2024-08-19T15:54:06.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:06.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj949 testdata.536210 2024-08-19T15:54:06.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:06.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj950 testdata.536210 2024-08-19T15:54:06.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:06.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj951 testdata.536210 2024-08-19T15:54:06.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:06.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj952 testdata.536210 2024-08-19T15:54:06.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:06.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj953 testdata.536210 2024-08-19T15:54:06.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:06.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj954 testdata.536210 2024-08-19T15:54:06.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:06.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj955 testdata.536210 2024-08-19T15:54:06.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:06.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj956 testdata.536210 2024-08-19T15:54:07.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:07.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj957 testdata.536210 2024-08-19T15:54:07.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:07.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj958 testdata.536210 2024-08-19T15:54:07.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:07.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj959 testdata.536210 2024-08-19T15:54:07.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:07.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj960 testdata.536210 2024-08-19T15:54:07.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:07.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj961 testdata.536210 2024-08-19T15:54:07.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:07.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj962 testdata.536210 2024-08-19T15:54:07.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:07.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj963 testdata.536210 2024-08-19T15:54:08.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:08.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj964 testdata.536210 2024-08-19T15:54:08.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:08.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj965 testdata.536210 2024-08-19T15:54:08.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:08.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj966 testdata.536210 2024-08-19T15:54:08.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:08.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj967 testdata.536210 2024-08-19T15:54:08.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:08.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj968 testdata.536210 2024-08-19T15:54:08.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:08.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj969 testdata.536210 2024-08-19T15:54:08.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:08.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj970 testdata.536210 2024-08-19T15:54:09.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:09.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj971 testdata.536210 2024-08-19T15:54:09.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:09.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj972 testdata.536210 2024-08-19T15:54:09.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:09.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj973 testdata.536210 2024-08-19T15:54:09.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:09.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj974 testdata.536210 2024-08-19T15:54:09.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:09.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj975 testdata.536210 2024-08-19T15:54:09.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:09.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj976 testdata.536210 2024-08-19T15:54:09.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:09.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj977 testdata.536210 2024-08-19T15:54:09.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:09.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj978 testdata.536210 2024-08-19T15:54:10.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:10.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj979 testdata.536210 2024-08-19T15:54:10.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:10.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj980 testdata.536210 2024-08-19T15:54:10.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:10.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj981 testdata.536210 2024-08-19T15:54:10.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:10.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj982 testdata.536210 2024-08-19T15:54:10.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:10.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj983 testdata.536210 2024-08-19T15:54:10.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:10.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj984 testdata.536210 2024-08-19T15:54:10.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:10.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj985 testdata.536210 2024-08-19T15:54:11.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:11.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj986 testdata.536210 2024-08-19T15:54:11.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:11.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj987 testdata.536210 2024-08-19T15:54:11.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:11.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj988 testdata.536210 2024-08-19T15:54:11.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:11.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj989 testdata.536210 2024-08-19T15:54:11.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:11.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj990 testdata.536210 2024-08-19T15:54:11.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:11.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj991 testdata.536210 2024-08-19T15:54:11.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:11.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj992 testdata.536210 2024-08-19T15:54:12.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:12.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj993 testdata.536210 2024-08-19T15:54:12.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:12.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj994 testdata.536210 2024-08-19T15:54:12.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:12.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj995 testdata.536210 2024-08-19T15:54:12.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:12.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj996 testdata.536210 2024-08-19T15:54:12.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:12.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj997 testdata.536210 2024-08-19T15:54:12.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:12.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj998 testdata.536210 2024-08-19T15:54:12.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:12.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj999 testdata.536210 2024-08-19T15:54:12.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-08-19T15:54:12.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj1000 testdata.536210 2024-08-19T15:54:13.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rm -f testdata.536210 2024-08-19T15:54:13.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:341: _scrub_abort: get_primary test obj1 2024-08-19T15:54:13.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-08-19T15:54:13.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-08-19T15:54:13.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-08-19T15:54:13.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T15:54:13.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:341: _scrub_abort: local primary=1 2024-08-19T15:54:13.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:342: _scrub_abort: local pgid=1.0 2024-08-19T15:54:13.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: ceph tell 1.0 schedule-deep-scrub 2024-08-19T15:54:13.398 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T15:54:13.398 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": true, 2024-08-19T15:54:13.398 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T15:54:13.399 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-05T15:52:33.397590+0000" 2024-08-19T15:54:13.399 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T15:54:13.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: set -o pipefail 2024-08-19T15:54:13.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:348: _scrub_abort: found=no 2024-08-19T15:54:13.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: seq 0 200 2024-08-19T15:54:13.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:54:13.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: flush_pg_stats 2024-08-19T15:54:13.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:54:13.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:54:13.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:54:13.669 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:54:13.669 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:54:13.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:54:13.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:13.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:54:13.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836516 2024-08-19T15:54:13.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836516 2024-08-19T15:54:13.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516' 2024-08-19T15:54:13.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:13.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:54:13.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672994 2024-08-19T15:54:13.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672994 2024-08-19T15:54:13.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672994' 2024-08-19T15:54:13.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:13.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:54:13.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509472 2024-08-19T15:54:13.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509472 2024-08-19T15:54:13.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672994 2-64424509472' 2024-08-19T15:54:13.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:13.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836516 2024-08-19T15:54:13.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:13.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:54:13.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836516 2024-08-19T15:54:13.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:13.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836516 2024-08-19T15:54:13.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-08-19T15:54:13.973 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836516 2024-08-19T15:54:13.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:14.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836515 -lt 21474836516 2024-08-19T15:54:14.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:54:15.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:54:15.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:15.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836516 2024-08-19T15:54:15.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:15.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672994 2024-08-19T15:54:15.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:15.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:54:15.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672994 2024-08-19T15:54:15.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:15.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672994 2024-08-19T15:54:15.512 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672994 2024-08-19T15:54:15.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2024-08-19T15:54:15.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:54:15.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672994 -lt 42949672994 2024-08-19T15:54:15.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:15.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509472 2024-08-19T15:54:15.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:15.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:54:15.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509472 2024-08-19T15:54:15.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:15.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509472 2024-08-19T15:54:15.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2024-08-19T15:54:15.788 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509472 2024-08-19T15:54:15.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:54:16.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509472 -lt 64424509472 2024-08-19T15:54:16.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: ceph pg dump pgs 2024-08-19T15:54:16.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: grep '^1.0' 2024-08-19T15:54:16.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: grep -q scrubbing 2024-08-19T15:54:16.342 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:54:16.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:54:16.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: flush_pg_stats 2024-08-19T15:54:16.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:54:16.354 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:54:16.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:54:16.617 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:54:16.617 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:54:16.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:54:16.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:16.617 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:54:16.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836518 2024-08-19T15:54:16.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836518 2024-08-19T15:54:16.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518' 2024-08-19T15:54:16.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:16.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:54:16.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672996 2024-08-19T15:54:16.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672996 2024-08-19T15:54:16.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-42949672996' 2024-08-19T15:54:16.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:16.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:54:16.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509473 2024-08-19T15:54:16.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509473 2024-08-19T15:54:16.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-42949672996 2-64424509473' 2024-08-19T15:54:16.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:16.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836518 2024-08-19T15:54:16.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:16.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:54:16.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836518 2024-08-19T15:54:16.920 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:16.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836518 2024-08-19T15:54:16.921 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836518 2024-08-19T15:54:16.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-08-19T15:54:16.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:17.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836518 -lt 21474836518 2024-08-19T15:54:17.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:17.196 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672996 2024-08-19T15:54:17.196 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:17.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:54:17.197 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672996 2024-08-19T15:54:17.197 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:17.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672996 2024-08-19T15:54:17.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-08-19T15:54:17.199 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672996 2024-08-19T15:54:17.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:54:17.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672996 -lt 42949672996 2024-08-19T15:54:17.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:17.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509473 2024-08-19T15:54:17.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:17.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:54:17.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509473 2024-08-19T15:54:17.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:17.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509473 2024-08-19T15:54:17.490 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509473 2024-08-19T15:54:17.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2024-08-19T15:54:17.490 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:54:17.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509473 -lt 64424509473 2024-08-19T15:54:17.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: ceph pg dump pgs 2024-08-19T15:54:17.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: grep '^1.0' 2024-08-19T15:54:17.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: grep -q scrubbing 2024-08-19T15:54:18.003 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:54:18.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:54:18.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: flush_pg_stats 2024-08-19T15:54:18.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:54:18.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:54:18.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:54:18.276 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:54:18.277 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:54:18.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:54:18.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:18.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:54:18.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836519 2024-08-19T15:54:18.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836519 2024-08-19T15:54:18.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519' 2024-08-19T15:54:18.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:18.377 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:54:18.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672997 2024-08-19T15:54:18.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672997 2024-08-19T15:54:18.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-42949672997' 2024-08-19T15:54:18.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:18.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:54:18.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509475 2024-08-19T15:54:18.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509475 2024-08-19T15:54:18.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-42949672997 2-64424509475' 2024-08-19T15:54:18.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:18.577 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836519 2024-08-19T15:54:18.577 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:18.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:54:18.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836519 2024-08-19T15:54:18.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:18.579 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836519 2024-08-19T15:54:18.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836519 2024-08-19T15:54:18.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2024-08-19T15:54:18.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:18.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836518 -lt 21474836519 2024-08-19T15:54:18.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:54:19.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:54:19.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:20.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836519 -lt 21474836519 2024-08-19T15:54:20.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:20.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672997 2024-08-19T15:54:20.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:20.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:54:20.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672997 2024-08-19T15:54:20.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:20.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672997 2024-08-19T15:54:20.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672997' 2024-08-19T15:54:20.122 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672997 2024-08-19T15:54:20.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:54:20.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672997 -lt 42949672997 2024-08-19T15:54:20.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:20.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509475 2024-08-19T15:54:20.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:20.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:54:20.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509475 2024-08-19T15:54:20.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:20.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509475 2024-08-19T15:54:20.388 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509475 2024-08-19T15:54:20.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2024-08-19T15:54:20.388 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:54:20.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509475 -lt 64424509475 2024-08-19T15:54:20.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: ceph pg dump pgs 2024-08-19T15:54:20.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: grep '^1.0' 2024-08-19T15:54:20.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: grep -q scrubbing 2024-08-19T15:54:21.047 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:54:21.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:54:21.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: flush_pg_stats 2024-08-19T15:54:21.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:54:21.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:54:21.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:54:21.321 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:54:21.321 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:54:21.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:54:21.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:21.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:54:21.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836521 2024-08-19T15:54:21.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836521 2024-08-19T15:54:21.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521' 2024-08-19T15:54:21.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:21.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:54:21.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672998 2024-08-19T15:54:21.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672998 2024-08-19T15:54:21.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-42949672998' 2024-08-19T15:54:21.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:21.522 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:54:21.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509476 2024-08-19T15:54:21.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509476 2024-08-19T15:54:21.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-42949672998 2-64424509476' 2024-08-19T15:54:21.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:21.622 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836521 2024-08-19T15:54:21.622 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:21.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:54:21.623 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836521 2024-08-19T15:54:21.623 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:21.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836521 2024-08-19T15:54:21.624 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836521 2024-08-19T15:54:21.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-08-19T15:54:21.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:21.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836520 -lt 21474836521 2024-08-19T15:54:21.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:54:22.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:54:22.887 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:23.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836521 -lt 21474836521 2024-08-19T15:54:23.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:23.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672998 2024-08-19T15:54:23.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:23.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:54:23.164 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672998 2024-08-19T15:54:23.164 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:23.165 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672998 2024-08-19T15:54:23.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672998 2024-08-19T15:54:23.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-08-19T15:54:23.166 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:54:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672999 -lt 42949672998 2024-08-19T15:54:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:23.430 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509476 2024-08-19T15:54:23.430 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:23.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:54:23.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509476 2024-08-19T15:54:23.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:23.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509476 2024-08-19T15:54:23.432 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509476 2024-08-19T15:54:23.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509476' 2024-08-19T15:54:23.433 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:54:23.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509477 -lt 64424509476 2024-08-19T15:54:23.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: ceph pg dump pgs 2024-08-19T15:54:23.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: grep '^1.0' 2024-08-19T15:54:23.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: grep -q scrubbing 2024-08-19T15:54:23.960 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:54:23.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: found=yes 2024-08-19T15:54:23.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:356: _scrub_abort: break 2024-08-19T15:54:23.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:359: _scrub_abort: set +o pipefail 2024-08-19T15:54:23.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: test yes = no 2024-08-19T15:54:23.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:367: _scrub_abort: ceph osd set nodeep-scrub 2024-08-19T15:54:24.360 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T15:54:24.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:368: _scrub_abort: '[' deep-scrub = deep-scrub ']' 2024-08-19T15:54:24.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: ceph osd set noscrub 2024-08-19T15:54:24.737 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T15:54:24.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:374: _scrub_abort: set -o pipefail 2024-08-19T15:54:24.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: seq 0 200 2024-08-19T15:54:24.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:54:24.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:54:24.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:54:24.750 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:54:25.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:54:25.034 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:54:25.034 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:54:25.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:54:25.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:25.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:54:25.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836522 2024-08-19T15:54:25.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836522 2024-08-19T15:54:25.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522' 2024-08-19T15:54:25.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:25.136 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:54:25.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673000 2024-08-19T15:54:25.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673000 2024-08-19T15:54:25.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522 1-42949673000' 2024-08-19T15:54:25.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:25.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:54:25.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509478 2024-08-19T15:54:25.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509478 2024-08-19T15:54:25.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522 1-42949673000 2-64424509478' 2024-08-19T15:54:25.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:25.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836522 2024-08-19T15:54:25.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:25.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:54:25.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836522 2024-08-19T15:54:25.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:25.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836522 2024-08-19T15:54:25.371 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836522 2024-08-19T15:54:25.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2024-08-19T15:54:25.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:25.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836521 -lt 21474836522 2024-08-19T15:54:25.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:54:26.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:54:26.633 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:26.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836521 -lt 21474836522 2024-08-19T15:54:26.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:54:27.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:54:27.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:28.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836523 -lt 21474836522 2024-08-19T15:54:28.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:28.182 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673000 2024-08-19T15:54:28.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:28.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:54:28.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673000 2024-08-19T15:54:28.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:28.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673000 2024-08-19T15:54:28.184 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673000 2024-08-19T15:54:28.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673000' 2024-08-19T15:54:28.185 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:54:28.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673001 -lt 42949673000 2024-08-19T15:54:28.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:28.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509478 2024-08-19T15:54:28.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:28.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:54:28.449 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509478 2024-08-19T15:54:28.449 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:28.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509478 2024-08-19T15:54:28.450 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509478 2024-08-19T15:54:28.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2024-08-19T15:54:28.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:54:28.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509478 -lt 64424509478 2024-08-19T15:54:28.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:54:28.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:54:28.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:54:28.962 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:54:28.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:54:28.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:54:28.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:54:28.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:54:28.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:54:29.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:54:29.237 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:54:29.237 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:54:29.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:54:29.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:29.238 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:54:29.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836524 2024-08-19T15:54:29.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836524 2024-08-19T15:54:29.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524' 2024-08-19T15:54:29.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:29.339 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:54:29.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673002 2024-08-19T15:54:29.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673002 2024-08-19T15:54:29.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-42949673002' 2024-08-19T15:54:29.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:29.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:54:29.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509480 2024-08-19T15:54:29.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509480 2024-08-19T15:54:29.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509480' 2024-08-19T15:54:29.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:29.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836524 2024-08-19T15:54:29.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:29.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:54:29.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836524 2024-08-19T15:54:29.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:29.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836524 2024-08-19T15:54:29.542 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836524 2024-08-19T15:54:29.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-08-19T15:54:29.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:29.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836523 -lt 21474836524 2024-08-19T15:54:29.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:54:30.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:54:30.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:31.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836524 2024-08-19T15:54:31.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:31.100 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673002 2024-08-19T15:54:31.100 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:31.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:54:31.101 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673002 2024-08-19T15:54:31.101 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:31.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673002 2024-08-19T15:54:31.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673002' 2024-08-19T15:54:31.103 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673002 2024-08-19T15:54:31.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:54:31.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673002 -lt 42949673002 2024-08-19T15:54:31.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:31.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509480 2024-08-19T15:54:31.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:31.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:54:31.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509480 2024-08-19T15:54:31.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:31.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509480 2024-08-19T15:54:31.371 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509480 2024-08-19T15:54:31.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509480' 2024-08-19T15:54:31.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:54:31.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509480 -lt 64424509480 2024-08-19T15:54:31.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:54:31.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:54:31.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:54:31.891 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:54:31.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:54:31.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:54:31.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:54:31.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:54:31.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:54:32.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:54:32.164 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:54:32.165 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:54:32.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:54:32.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:32.165 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:54:32.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836526 2024-08-19T15:54:32.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836526 2024-08-19T15:54:32.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836526' 2024-08-19T15:54:32.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:32.265 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:54:32.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673004 2024-08-19T15:54:32.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673004 2024-08-19T15:54:32.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836526 1-42949673004' 2024-08-19T15:54:32.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:32.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:54:32.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509481 2024-08-19T15:54:32.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509481 2024-08-19T15:54:32.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836526 1-42949673004 2-64424509481' 2024-08-19T15:54:32.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:32.466 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836526 2024-08-19T15:54:32.466 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:32.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:54:32.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836526 2024-08-19T15:54:32.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:32.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836526 2024-08-19T15:54:32.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-08-19T15:54:32.469 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836526 2024-08-19T15:54:32.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:32.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836526 2024-08-19T15:54:32.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:54:33.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:54:33.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:34.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836526 -lt 21474836526 2024-08-19T15:54:34.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:34.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673004 2024-08-19T15:54:34.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:34.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:54:34.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673004 2024-08-19T15:54:34.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:34.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673004 2024-08-19T15:54:34.005 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673004 2024-08-19T15:54:34.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673004' 2024-08-19T15:54:34.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:54:34.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673004 -lt 42949673004 2024-08-19T15:54:34.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:34.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509481 2024-08-19T15:54:34.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:34.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:54:34.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509481 2024-08-19T15:54:34.269 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:34.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509481 2024-08-19T15:54:34.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509481' 2024-08-19T15:54:34.270 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509481 2024-08-19T15:54:34.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:54:34.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509482 -lt 64424509481 2024-08-19T15:54:34.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:54:34.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:54:34.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:54:34.785 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:54:34.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:54:34.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:54:34.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:54:34.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:54:34.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:54:35.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:54:35.058 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:54:35.058 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:54:35.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:54:35.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:35.058 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:54:35.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836527 2024-08-19T15:54:35.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836527 2024-08-19T15:54:35.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527' 2024-08-19T15:54:35.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:35.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:54:35.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673005 2024-08-19T15:54:35.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673005 2024-08-19T15:54:35.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-42949673005' 2024-08-19T15:54:35.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:35.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:54:35.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509483 2024-08-19T15:54:35.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509483 2024-08-19T15:54:35.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-42949673005 2-64424509483' 2024-08-19T15:54:35.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:35.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836527 2024-08-19T15:54:35.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:35.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:54:35.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836527 2024-08-19T15:54:35.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:35.376 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836527 2024-08-19T15:54:35.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836527 2024-08-19T15:54:35.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-08-19T15:54:35.377 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:35.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836526 -lt 21474836527 2024-08-19T15:54:35.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:54:36.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:54:36.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:36.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836526 -lt 21474836527 2024-08-19T15:54:36.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:54:37.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:54:37.920 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:38.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836528 -lt 21474836527 2024-08-19T15:54:38.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:38.186 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673005 2024-08-19T15:54:38.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:38.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:54:38.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673005 2024-08-19T15:54:38.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:38.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673005 2024-08-19T15:54:38.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673005' 2024-08-19T15:54:38.189 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673005 2024-08-19T15:54:38.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:54:38.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673006 -lt 42949673005 2024-08-19T15:54:38.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:38.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509483 2024-08-19T15:54:38.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:38.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:54:38.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509483 2024-08-19T15:54:38.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:38.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509483 2024-08-19T15:54:38.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509483' 2024-08-19T15:54:38.460 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509483 2024-08-19T15:54:38.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:54:38.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509483 -lt 64424509483 2024-08-19T15:54:38.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:54:38.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:54:38.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:54:38.972 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:54:38.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:54:38.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:54:38.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:54:38.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:54:38.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:54:39.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:54:39.249 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:54:39.249 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:54:39.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:54:39.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:39.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:54:39.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836529 2024-08-19T15:54:39.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836529 2024-08-19T15:54:39.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529' 2024-08-19T15:54:39.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:39.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:54:39.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673007 2024-08-19T15:54:39.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673007 2024-08-19T15:54:39.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529 1-42949673007' 2024-08-19T15:54:39.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:39.448 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:54:39.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509485 2024-08-19T15:54:39.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509485 2024-08-19T15:54:39.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529 1-42949673007 2-64424509485' 2024-08-19T15:54:39.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:39.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836529 2024-08-19T15:54:39.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:39.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:54:39.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836529 2024-08-19T15:54:39.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:39.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836529 2024-08-19T15:54:39.551 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836529 2024-08-19T15:54:39.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2024-08-19T15:54:39.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:39.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836528 -lt 21474836529 2024-08-19T15:54:39.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:54:40.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:54:40.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:41.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836528 -lt 21474836529 2024-08-19T15:54:41.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:54:42.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:54:42.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:42.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836530 -lt 21474836529 2024-08-19T15:54:42.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:42.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673007 2024-08-19T15:54:42.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:42.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:54:42.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673007 2024-08-19T15:54:42.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:42.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673007 2024-08-19T15:54:42.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673007' 2024-08-19T15:54:42.460 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673007 2024-08-19T15:54:42.461 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:54:42.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673007 -lt 42949673007 2024-08-19T15:54:42.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:42.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509485 2024-08-19T15:54:42.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:42.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:54:42.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509485 2024-08-19T15:54:42.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:42.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509485 2024-08-19T15:54:42.729 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509485 2024-08-19T15:54:42.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509485' 2024-08-19T15:54:42.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:54:42.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509485 -lt 64424509485 2024-08-19T15:54:42.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:54:42.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:54:42.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:54:43.244 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:54:43.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:54:43.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:54:43.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:54:43.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:54:43.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:54:43.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:54:43.517 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:54:43.517 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:54:43.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:54:43.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:43.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:54:43.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836531 2024-08-19T15:54:43.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836531 2024-08-19T15:54:43.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531' 2024-08-19T15:54:43.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:43.616 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:54:43.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673009 2024-08-19T15:54:43.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673009 2024-08-19T15:54:43.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-42949673009' 2024-08-19T15:54:43.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:43.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:54:43.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509487 2024-08-19T15:54:43.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509487 2024-08-19T15:54:43.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-42949673009 2-64424509487' 2024-08-19T15:54:43.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:43.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836531 2024-08-19T15:54:43.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:43.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:54:43.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836531 2024-08-19T15:54:43.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:43.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836531 2024-08-19T15:54:43.822 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836531 2024-08-19T15:54:43.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2024-08-19T15:54:43.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:44.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836530 -lt 21474836531 2024-08-19T15:54:44.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:54:45.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:54:45.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:45.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836531 -lt 21474836531 2024-08-19T15:54:45.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:45.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673009 2024-08-19T15:54:45.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:45.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:54:45.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673009 2024-08-19T15:54:45.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:45.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673009 2024-08-19T15:54:45.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673009' 2024-08-19T15:54:45.360 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673009 2024-08-19T15:54:45.360 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:54:45.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673009 -lt 42949673009 2024-08-19T15:54:45.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:45.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509487 2024-08-19T15:54:45.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:45.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:54:45.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509487 2024-08-19T15:54:45.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:45.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509487 2024-08-19T15:54:45.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509487' 2024-08-19T15:54:45.628 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509487 2024-08-19T15:54:45.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:54:45.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509487 -lt 64424509487 2024-08-19T15:54:45.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:54:45.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:54:45.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:54:46.141 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:54:46.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:54:46.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:54:46.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:54:46.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:54:46.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:54:46.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:54:46.414 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:54:46.414 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:54:46.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:54:46.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:46.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:54:46.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836533 2024-08-19T15:54:46.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836533 2024-08-19T15:54:46.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533' 2024-08-19T15:54:46.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:46.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:54:46.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673010 2024-08-19T15:54:46.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673010 2024-08-19T15:54:46.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533 1-42949673010' 2024-08-19T15:54:46.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:46.617 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:54:46.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509488 2024-08-19T15:54:46.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509488 2024-08-19T15:54:46.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533 1-42949673010 2-64424509488' 2024-08-19T15:54:46.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:46.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836533 2024-08-19T15:54:46.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:46.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:54:46.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836533 2024-08-19T15:54:46.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:46.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836533 2024-08-19T15:54:46.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2024-08-19T15:54:46.719 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836533 2024-08-19T15:54:46.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:46.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836531 -lt 21474836533 2024-08-19T15:54:46.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:54:47.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:54:47.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:48.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836533 -lt 21474836533 2024-08-19T15:54:48.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:48.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673010 2024-08-19T15:54:48.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:48.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:54:48.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673010 2024-08-19T15:54:48.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:48.256 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673010 2024-08-19T15:54:48.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673010 2024-08-19T15:54:48.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673010' 2024-08-19T15:54:48.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:54:48.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673011 -lt 42949673010 2024-08-19T15:54:48.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:48.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509488 2024-08-19T15:54:48.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:48.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:54:48.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509488 2024-08-19T15:54:48.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:48.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509488 2024-08-19T15:54:48.522 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509488 2024-08-19T15:54:48.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509488' 2024-08-19T15:54:48.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:54:48.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509488 -lt 64424509488 2024-08-19T15:54:48.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:54:48.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:54:48.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:54:49.037 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:54:49.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:54:49.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:54:49.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:54:49.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:54:49.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:54:49.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:54:49.330 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:54:49.330 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:54:49.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:54:49.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:49.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:54:49.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836534 2024-08-19T15:54:49.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836534 2024-08-19T15:54:49.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534' 2024-08-19T15:54:49.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:49.430 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:54:49.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673012 2024-08-19T15:54:49.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673012 2024-08-19T15:54:49.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-42949673012' 2024-08-19T15:54:49.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:49.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:54:49.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509490 2024-08-19T15:54:49.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509490 2024-08-19T15:54:49.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509490' 2024-08-19T15:54:49.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:49.631 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836534 2024-08-19T15:54:49.631 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:49.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:54:49.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836534 2024-08-19T15:54:49.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:49.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836534 2024-08-19T15:54:49.633 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836534 2024-08-19T15:54:49.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-08-19T15:54:49.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:49.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836533 -lt 21474836534 2024-08-19T15:54:49.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:54:50.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:54:50.897 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:51.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836535 -lt 21474836534 2024-08-19T15:54:51.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:51.166 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673012 2024-08-19T15:54:51.166 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:51.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:54:51.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673012 2024-08-19T15:54:51.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:51.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673012 2024-08-19T15:54:51.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673012' 2024-08-19T15:54:51.171 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673012 2024-08-19T15:54:51.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:54:51.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673012 -lt 42949673012 2024-08-19T15:54:51.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:51.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509490 2024-08-19T15:54:51.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:51.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:54:51.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509490 2024-08-19T15:54:51.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:51.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509490 2024-08-19T15:54:51.460 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509490 2024-08-19T15:54:51.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509490' 2024-08-19T15:54:51.461 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:54:51.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509490 -lt 64424509490 2024-08-19T15:54:51.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:54:51.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:54:51.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:54:51.976 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:54:51.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:54:51.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:54:51.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:54:51.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:54:51.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:54:52.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:54:52.267 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:54:52.267 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:54:52.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:54:52.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:52.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:54:52.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836536 2024-08-19T15:54:52.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836536 2024-08-19T15:54:52.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536' 2024-08-19T15:54:52.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:52.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:54:52.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673014 2024-08-19T15:54:52.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673014 2024-08-19T15:54:52.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-42949673014' 2024-08-19T15:54:52.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:52.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:54:52.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509491 2024-08-19T15:54:52.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509491 2024-08-19T15:54:52.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-42949673014 2-64424509491' 2024-08-19T15:54:52.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:52.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836536 2024-08-19T15:54:52.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:52.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:54:52.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836536 2024-08-19T15:54:52.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:52.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836536 2024-08-19T15:54:52.570 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836536 2024-08-19T15:54:52.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2024-08-19T15:54:52.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:52.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836535 -lt 21474836536 2024-08-19T15:54:52.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:54:53.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:54:53.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:54.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836536 2024-08-19T15:54:54.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:54.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673014 2024-08-19T15:54:54.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:54.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:54:54.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673014 2024-08-19T15:54:54.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:54.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673014 2024-08-19T15:54:54.107 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673014 2024-08-19T15:54:54.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673014' 2024-08-19T15:54:54.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:54:54.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673014 -lt 42949673014 2024-08-19T15:54:54.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:54.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509491 2024-08-19T15:54:54.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:54.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:54:54.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509491 2024-08-19T15:54:54.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:54.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509491 2024-08-19T15:54:54.379 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509491 2024-08-19T15:54:54.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509491' 2024-08-19T15:54:54.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:54:54.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509492 -lt 64424509491 2024-08-19T15:54:54.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:54:54.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:54:54.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:54:54.890 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:54:54.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:54:54.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:54:54.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:54:54.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:54:54.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:54:55.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:54:55.167 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:54:55.167 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:54:55.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:54:55.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:55.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:54:55.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836537 2024-08-19T15:54:55.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836537 2024-08-19T15:54:55.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537' 2024-08-19T15:54:55.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:55.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:54:55.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673015 2024-08-19T15:54:55.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673015 2024-08-19T15:54:55.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-42949673015' 2024-08-19T15:54:55.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:55.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:54:55.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509493 2024-08-19T15:54:55.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509493 2024-08-19T15:54:55.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-42949673015 2-64424509493' 2024-08-19T15:54:55.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:55.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836537 2024-08-19T15:54:55.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:55.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:54:55.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836537 2024-08-19T15:54:55.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:55.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836537 2024-08-19T15:54:55.470 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836537 2024-08-19T15:54:55.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-08-19T15:54:55.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:55.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836537 2024-08-19T15:54:55.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:54:56.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:54:56.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:57.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836537 2024-08-19T15:54:57.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:54:58.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:54:58.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:58.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836537 2024-08-19T15:54:58.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:58.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673015 2024-08-19T15:54:58.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:58.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:54:58.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673015 2024-08-19T15:54:58.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:58.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673015 2024-08-19T15:54:58.309 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673015 2024-08-19T15:54:58.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673015' 2024-08-19T15:54:58.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:54:58.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673016 -lt 42949673015 2024-08-19T15:54:58.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:58.574 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509493 2024-08-19T15:54:58.574 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:58.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:54:58.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509493 2024-08-19T15:54:58.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:58.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509493 2024-08-19T15:54:58.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509493' 2024-08-19T15:54:58.576 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509493 2024-08-19T15:54:58.577 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:54:58.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509493 -lt 64424509493 2024-08-19T15:54:58.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:54:58.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:54:58.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:54:59.093 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:54:59.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:54:59.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:54:59.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:54:59.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:54:59.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:54:59.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:54:59.368 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:54:59.368 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:54:59.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:54:59.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:59.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:54:59.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836539 2024-08-19T15:54:59.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836539 2024-08-19T15:54:59.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539' 2024-08-19T15:54:59.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:59.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:54:59.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673017 2024-08-19T15:54:59.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673017 2024-08-19T15:54:59.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-42949673017' 2024-08-19T15:54:59.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:54:59.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:54:59.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509495 2024-08-19T15:54:59.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509495 2024-08-19T15:54:59.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-42949673017 2-64424509495' 2024-08-19T15:54:59.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:54:59.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836539 2024-08-19T15:54:59.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:54:59.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:54:59.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836539 2024-08-19T15:54:59.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:54:59.716 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836539 2024-08-19T15:54:59.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836539 2024-08-19T15:54:59.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836539' 2024-08-19T15:54:59.717 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:54:59.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836539 2024-08-19T15:54:59.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:00.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:55:00.982 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:01.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836540 -lt 21474836539 2024-08-19T15:55:01.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:01.282 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673017 2024-08-19T15:55:01.282 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:01.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:55:01.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673017 2024-08-19T15:55:01.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:01.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673017 2024-08-19T15:55:01.285 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673017 2024-08-19T15:55:01.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673017' 2024-08-19T15:55:01.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:55:01.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673017 -lt 42949673017 2024-08-19T15:55:01.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:01.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509495 2024-08-19T15:55:01.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:01.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:55:01.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509495 2024-08-19T15:55:01.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:01.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509495 2024-08-19T15:55:01.554 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509495 2024-08-19T15:55:01.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509495' 2024-08-19T15:55:01.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:55:01.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509495 -lt 64424509495 2024-08-19T15:55:01.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:55:01.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:55:01.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:55:02.066 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:55:02.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:55:02.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:55:02.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:55:02.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:55:02.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:55:02.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:55:02.340 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:55:02.340 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:55:02.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:55:02.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:02.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:55:02.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836541 2024-08-19T15:55:02.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836541 2024-08-19T15:55:02.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541' 2024-08-19T15:55:02.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:02.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:55:02.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673019 2024-08-19T15:55:02.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673019 2024-08-19T15:55:02.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-42949673019' 2024-08-19T15:55:02.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:02.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:55:02.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509496 2024-08-19T15:55:02.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509496 2024-08-19T15:55:02.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-42949673019 2-64424509496' 2024-08-19T15:55:02.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:02.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836541 2024-08-19T15:55:02.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:02.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:55:02.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836541 2024-08-19T15:55:02.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:02.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836541 2024-08-19T15:55:02.643 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836541 2024-08-19T15:55:02.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2024-08-19T15:55:02.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:02.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836540 -lt 21474836541 2024-08-19T15:55:02.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:03.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:55:03.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:04.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836541 2024-08-19T15:55:04.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:04.185 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673019 2024-08-19T15:55:04.185 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:04.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:55:04.186 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673019 2024-08-19T15:55:04.186 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:04.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673019 2024-08-19T15:55:04.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673019' 2024-08-19T15:55:04.187 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673019 2024-08-19T15:55:04.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:55:04.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673019 -lt 42949673019 2024-08-19T15:55:04.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:04.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509496 2024-08-19T15:55:04.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:04.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:55:04.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509496 2024-08-19T15:55:04.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:04.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509496 2024-08-19T15:55:04.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509496' 2024-08-19T15:55:04.457 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509496 2024-08-19T15:55:04.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:55:04.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509497 -lt 64424509496 2024-08-19T15:55:04.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:55:04.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:55:04.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:55:04.970 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:55:04.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:55:04.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:55:04.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:55:04.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:55:04.982 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:55:05.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:55:05.247 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:55:05.247 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:55:05.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:55:05.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:05.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:55:05.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836542 2024-08-19T15:55:05.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836542 2024-08-19T15:55:05.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542' 2024-08-19T15:55:05.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:05.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:55:05.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673020 2024-08-19T15:55:05.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673020 2024-08-19T15:55:05.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-42949673020' 2024-08-19T15:55:05.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:05.449 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:55:05.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509498 2024-08-19T15:55:05.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509498 2024-08-19T15:55:05.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-42949673020 2-64424509498' 2024-08-19T15:55:05.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:05.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836542 2024-08-19T15:55:05.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:05.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:55:05.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836542 2024-08-19T15:55:05.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:05.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836542 2024-08-19T15:55:05.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836542' 2024-08-19T15:55:05.552 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836542 2024-08-19T15:55:05.552 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:05.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836542 2024-08-19T15:55:05.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:06.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:55:06.816 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:07.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836543 -lt 21474836542 2024-08-19T15:55:07.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:07.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673020 2024-08-19T15:55:07.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:07.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:55:07.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673020 2024-08-19T15:55:07.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:07.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673020 2024-08-19T15:55:07.108 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673020 2024-08-19T15:55:07.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673020' 2024-08-19T15:55:07.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:55:07.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673021 -lt 42949673020 2024-08-19T15:55:07.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:07.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509498 2024-08-19T15:55:07.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:07.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:55:07.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509498 2024-08-19T15:55:07.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:07.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509498 2024-08-19T15:55:07.377 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509498 2024-08-19T15:55:07.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509498' 2024-08-19T15:55:07.377 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:55:07.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509498 -lt 64424509498 2024-08-19T15:55:07.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:55:07.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:55:07.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:55:07.900 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:55:07.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:55:07.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:55:07.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:55:07.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:55:07.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:55:08.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:55:08.174 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:55:08.174 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:55:08.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:55:08.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:08.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:55:08.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836544 2024-08-19T15:55:08.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836544 2024-08-19T15:55:08.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544' 2024-08-19T15:55:08.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:08.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:55:08.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673022 2024-08-19T15:55:08.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673022 2024-08-19T15:55:08.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673022' 2024-08-19T15:55:08.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:08.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:55:08.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509500 2024-08-19T15:55:08.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509500 2024-08-19T15:55:08.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673022 2-64424509500' 2024-08-19T15:55:08.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:08.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836544 2024-08-19T15:55:08.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:08.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:55:08.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836544 2024-08-19T15:55:08.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:08.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836544 2024-08-19T15:55:08.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836544' 2024-08-19T15:55:08.478 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836544 2024-08-19T15:55:08.479 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:08.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836543 -lt 21474836544 2024-08-19T15:55:08.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:09.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:55:09.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:10.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836544 -lt 21474836544 2024-08-19T15:55:10.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:10.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673022 2024-08-19T15:55:10.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:10.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:55:10.020 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673022 2024-08-19T15:55:10.020 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:10.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673022 2024-08-19T15:55:10.021 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673022 2024-08-19T15:55:10.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673022' 2024-08-19T15:55:10.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:55:10.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673022 -lt 42949673022 2024-08-19T15:55:10.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:10.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509500 2024-08-19T15:55:10.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:10.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:55:10.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509500 2024-08-19T15:55:10.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:10.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509500 2024-08-19T15:55:10.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509500' 2024-08-19T15:55:10.287 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509500 2024-08-19T15:55:10.287 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:55:10.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509500 -lt 64424509500 2024-08-19T15:55:10.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:55:10.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:55:10.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:55:10.800 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:55:10.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:55:10.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:55:10.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:55:10.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:55:10.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:55:11.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:55:11.076 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:55:11.076 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:55:11.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:55:11.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:11.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:55:11.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836546 2024-08-19T15:55:11.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836546 2024-08-19T15:55:11.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546' 2024-08-19T15:55:11.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:11.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:55:11.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673023 2024-08-19T15:55:11.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673023 2024-08-19T15:55:11.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673023' 2024-08-19T15:55:11.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:11.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:55:11.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509501 2024-08-19T15:55:11.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509501 2024-08-19T15:55:11.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673023 2-64424509501' 2024-08-19T15:55:11.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:11.388 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836546 2024-08-19T15:55:11.388 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:11.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:55:11.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836546 2024-08-19T15:55:11.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:11.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836546 2024-08-19T15:55:11.391 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836546 2024-08-19T15:55:11.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836546' 2024-08-19T15:55:11.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:11.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836545 -lt 21474836546 2024-08-19T15:55:11.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:12.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:55:12.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:12.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836545 -lt 21474836546 2024-08-19T15:55:12.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:13.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:55:13.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:14.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836546 -lt 21474836546 2024-08-19T15:55:14.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:14.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673023 2024-08-19T15:55:14.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:14.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:55:14.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673023 2024-08-19T15:55:14.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:14.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673023 2024-08-19T15:55:14.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673023' 2024-08-19T15:55:14.200 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673023 2024-08-19T15:55:14.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:55:14.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673024 -lt 42949673023 2024-08-19T15:55:14.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:14.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509501 2024-08-19T15:55:14.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:14.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:55:14.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509501 2024-08-19T15:55:14.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:14.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509501 2024-08-19T15:55:14.487 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509501 2024-08-19T15:55:14.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509501' 2024-08-19T15:55:14.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:55:14.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509502 -lt 64424509501 2024-08-19T15:55:14.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:55:14.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:55:14.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:55:15.000 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:55:15.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:55:15.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:55:15.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:55:15.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:55:15.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:55:15.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:55:15.286 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:55:15.286 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:55:15.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:55:15.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:15.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:55:15.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836547 2024-08-19T15:55:15.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836547 2024-08-19T15:55:15.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547' 2024-08-19T15:55:15.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:15.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:55:15.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673025 2024-08-19T15:55:15.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673025 2024-08-19T15:55:15.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673025' 2024-08-19T15:55:15.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:15.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:55:15.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509503 2024-08-19T15:55:15.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509503 2024-08-19T15:55:15.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673025 2-64424509503' 2024-08-19T15:55:15.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:15.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836547 2024-08-19T15:55:15.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:15.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:55:15.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836547 2024-08-19T15:55:15.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:15.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836547 2024-08-19T15:55:15.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2024-08-19T15:55:15.588 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836547 2024-08-19T15:55:15.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:15.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836546 -lt 21474836547 2024-08-19T15:55:15.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:16.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:55:16.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:17.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836548 -lt 21474836547 2024-08-19T15:55:17.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:17.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673025 2024-08-19T15:55:17.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:17.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:55:17.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673025 2024-08-19T15:55:17.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:17.157 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673025 2024-08-19T15:55:17.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673025 2024-08-19T15:55:17.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673025' 2024-08-19T15:55:17.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:55:17.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673026 -lt 42949673025 2024-08-19T15:55:17.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:17.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509503 2024-08-19T15:55:17.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:17.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:55:17.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509503 2024-08-19T15:55:17.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:17.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509503 2024-08-19T15:55:17.429 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509503 2024-08-19T15:55:17.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509503' 2024-08-19T15:55:17.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:55:17.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509503 -lt 64424509503 2024-08-19T15:55:17.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:55:17.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:55:17.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:55:17.941 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:55:17.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:55:17.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:55:17.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:55:17.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:55:17.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:55:18.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:55:18.215 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:55:18.215 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:55:18.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:55:18.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:18.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:55:18.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836549 2024-08-19T15:55:18.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836549 2024-08-19T15:55:18.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549' 2024-08-19T15:55:18.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:18.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:55:18.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673027 2024-08-19T15:55:18.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673027 2024-08-19T15:55:18.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673027' 2024-08-19T15:55:18.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:18.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:55:18.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509505 2024-08-19T15:55:18.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509505 2024-08-19T15:55:18.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673027 2-64424509505' 2024-08-19T15:55:18.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:18.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836549 2024-08-19T15:55:18.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:18.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:55:18.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836549 2024-08-19T15:55:18.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:18.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836549 2024-08-19T15:55:18.517 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836549 2024-08-19T15:55:18.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836549' 2024-08-19T15:55:18.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:18.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836548 -lt 21474836549 2024-08-19T15:55:18.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:19.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:55:19.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:20.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836549 -lt 21474836549 2024-08-19T15:55:20.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:20.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673027 2024-08-19T15:55:20.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:20.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:55:20.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673027 2024-08-19T15:55:20.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:20.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673027 2024-08-19T15:55:20.051 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673027 2024-08-19T15:55:20.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673027' 2024-08-19T15:55:20.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:55:20.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673027 -lt 42949673027 2024-08-19T15:55:20.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:20.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509505 2024-08-19T15:55:20.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:20.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:55:20.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509505 2024-08-19T15:55:20.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:20.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509505 2024-08-19T15:55:20.318 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509505 2024-08-19T15:55:20.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509505' 2024-08-19T15:55:20.319 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:55:20.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509505 -lt 64424509505 2024-08-19T15:55:20.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:55:20.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:55:20.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:55:20.832 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:55:20.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:55:20.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:55:20.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:55:20.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:55:20.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:55:21.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:55:21.109 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:55:21.109 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:55:21.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:55:21.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:21.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:55:21.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836551 2024-08-19T15:55:21.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836551 2024-08-19T15:55:21.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551' 2024-08-19T15:55:21.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:21.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:55:21.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673028 2024-08-19T15:55:21.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673028 2024-08-19T15:55:21.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673028' 2024-08-19T15:55:21.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:21.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:55:21.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509506 2024-08-19T15:55:21.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509506 2024-08-19T15:55:21.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673028 2-64424509506' 2024-08-19T15:55:21.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:21.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836551 2024-08-19T15:55:21.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:21.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:55:21.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836551 2024-08-19T15:55:21.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:21.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836551 2024-08-19T15:55:21.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836551' 2024-08-19T15:55:21.440 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836551 2024-08-19T15:55:21.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:21.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836550 -lt 21474836551 2024-08-19T15:55:21.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:22.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:55:22.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:22.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836550 -lt 21474836551 2024-08-19T15:55:22.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:23.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:55:23.974 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:24.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836551 -lt 21474836551 2024-08-19T15:55:24.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:24.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673028 2024-08-19T15:55:24.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:24.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:55:24.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673028 2024-08-19T15:55:24.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:24.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673028 2024-08-19T15:55:24.248 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673028 2024-08-19T15:55:24.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673028' 2024-08-19T15:55:24.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:55:24.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673029 -lt 42949673028 2024-08-19T15:55:24.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:24.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509506 2024-08-19T15:55:24.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:24.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:55:24.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509506 2024-08-19T15:55:24.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:24.521 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509506 2024-08-19T15:55:24.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509506 2024-08-19T15:55:24.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509506' 2024-08-19T15:55:24.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:55:24.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509507 -lt 64424509506 2024-08-19T15:55:24.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:55:24.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:55:24.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:55:25.034 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:55:25.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:55:25.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:55:25.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:55:25.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:55:25.046 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:55:25.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:55:25.352 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:55:25.352 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:55:25.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:55:25.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:25.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:55:25.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836553 2024-08-19T15:55:25.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836553 2024-08-19T15:55:25.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553' 2024-08-19T15:55:25.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:25.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:55:25.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673030 2024-08-19T15:55:25.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673030 2024-08-19T15:55:25.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673030' 2024-08-19T15:55:25.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:25.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:55:25.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509508 2024-08-19T15:55:25.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509508 2024-08-19T15:55:25.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673030 2-64424509508' 2024-08-19T15:55:25.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:25.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836553 2024-08-19T15:55:25.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:25.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:55:25.655 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836553 2024-08-19T15:55:25.655 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:25.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836553 2024-08-19T15:55:25.656 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836553 2024-08-19T15:55:25.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2024-08-19T15:55:25.656 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:25.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836551 -lt 21474836553 2024-08-19T15:55:25.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:26.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:55:26.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:27.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836553 -lt 21474836553 2024-08-19T15:55:27.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:27.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673030 2024-08-19T15:55:27.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:27.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:55:27.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673030 2024-08-19T15:55:27.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:27.256 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673030 2024-08-19T15:55:27.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673030 2024-08-19T15:55:27.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673030' 2024-08-19T15:55:27.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:55:27.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673031 -lt 42949673030 2024-08-19T15:55:27.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:27.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509508 2024-08-19T15:55:27.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:27.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:55:27.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509508 2024-08-19T15:55:27.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:27.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509508 2024-08-19T15:55:27.548 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509508 2024-08-19T15:55:27.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509508' 2024-08-19T15:55:27.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:55:27.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509508 -lt 64424509508 2024-08-19T15:55:27.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:55:27.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:55:27.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:55:28.063 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:55:28.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:55:28.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:55:28.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:55:28.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:55:28.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:55:28.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:55:28.335 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:55:28.335 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:55:28.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:55:28.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:28.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:55:28.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836554 2024-08-19T15:55:28.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836554 2024-08-19T15:55:28.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554' 2024-08-19T15:55:28.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:28.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:55:28.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673032 2024-08-19T15:55:28.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673032 2024-08-19T15:55:28.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673032' 2024-08-19T15:55:28.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:28.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:55:28.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509510 2024-08-19T15:55:28.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509510 2024-08-19T15:55:28.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673032 2-64424509510' 2024-08-19T15:55:28.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:28.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836554 2024-08-19T15:55:28.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:28.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:55:28.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836554 2024-08-19T15:55:28.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:28.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836554 2024-08-19T15:55:28.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836554' 2024-08-19T15:55:28.641 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836554 2024-08-19T15:55:28.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:28.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836553 -lt 21474836554 2024-08-19T15:55:28.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:29.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:55:29.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:30.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836554 -lt 21474836554 2024-08-19T15:55:30.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:30.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673032 2024-08-19T15:55:30.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:30.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:55:30.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673032 2024-08-19T15:55:30.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:30.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673032 2024-08-19T15:55:30.177 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673032 2024-08-19T15:55:30.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673032' 2024-08-19T15:55:30.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:55:30.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673032 -lt 42949673032 2024-08-19T15:55:30.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:30.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509510 2024-08-19T15:55:30.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:30.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:55:30.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509510 2024-08-19T15:55:30.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:30.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509510 2024-08-19T15:55:30.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509510' 2024-08-19T15:55:30.444 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509510 2024-08-19T15:55:30.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:55:30.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509510 -lt 64424509510 2024-08-19T15:55:30.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:55:30.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:55:30.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:55:30.955 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:55:30.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:55:30.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:55:30.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:55:30.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:55:30.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:55:31.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:55:31.230 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:55:31.231 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:55:31.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:55:31.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:31.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:55:31.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836556 2024-08-19T15:55:31.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836556 2024-08-19T15:55:31.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556' 2024-08-19T15:55:31.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:31.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:55:31.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673033 2024-08-19T15:55:31.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673033 2024-08-19T15:55:31.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673033' 2024-08-19T15:55:31.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:31.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:55:31.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509511 2024-08-19T15:55:31.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509511 2024-08-19T15:55:31.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673033 2-64424509511' 2024-08-19T15:55:31.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:31.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836556 2024-08-19T15:55:31.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:31.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:55:31.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836556 2024-08-19T15:55:31.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:31.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836556 2024-08-19T15:55:31.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836556' 2024-08-19T15:55:31.550 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836556 2024-08-19T15:55:31.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:31.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836555 -lt 21474836556 2024-08-19T15:55:31.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:32.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:55:32.813 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:33.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836556 -lt 21474836556 2024-08-19T15:55:33.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:33.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673033 2024-08-19T15:55:33.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:33.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:55:33.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673033 2024-08-19T15:55:33.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:33.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673033 2024-08-19T15:55:33.115 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673033 2024-08-19T15:55:33.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673033' 2024-08-19T15:55:33.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:55:33.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673034 -lt 42949673033 2024-08-19T15:55:33.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:33.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509511 2024-08-19T15:55:33.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:33.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:55:33.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509511 2024-08-19T15:55:33.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:33.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509511 2024-08-19T15:55:33.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509511' 2024-08-19T15:55:33.383 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509511 2024-08-19T15:55:33.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:55:33.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509512 -lt 64424509511 2024-08-19T15:55:33.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:55:33.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:55:33.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:55:33.905 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:55:33.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:55:33.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:55:33.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:55:33.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:55:33.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:55:34.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:55:34.180 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:55:34.180 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:55:34.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:55:34.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:34.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:55:34.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836557 2024-08-19T15:55:34.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836557 2024-08-19T15:55:34.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557' 2024-08-19T15:55:34.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:34.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:55:34.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673035 2024-08-19T15:55:34.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673035 2024-08-19T15:55:34.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673035' 2024-08-19T15:55:34.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:34.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:55:34.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509513 2024-08-19T15:55:34.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509513 2024-08-19T15:55:34.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673035 2-64424509513' 2024-08-19T15:55:34.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:34.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836557 2024-08-19T15:55:34.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:34.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:55:34.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836557 2024-08-19T15:55:34.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:34.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836557 2024-08-19T15:55:34.518 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836557 2024-08-19T15:55:34.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836557' 2024-08-19T15:55:34.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:34.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836556 -lt 21474836557 2024-08-19T15:55:34.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:35.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:55:35.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:36.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836557 -lt 21474836557 2024-08-19T15:55:36.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:36.045 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673035 2024-08-19T15:55:36.046 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:36.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:55:36.047 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673035 2024-08-19T15:55:36.047 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:36.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673035 2024-08-19T15:55:36.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673035' 2024-08-19T15:55:36.048 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673035 2024-08-19T15:55:36.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:55:36.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673035 -lt 42949673035 2024-08-19T15:55:36.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:36.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509513 2024-08-19T15:55:36.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:36.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:55:36.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509513 2024-08-19T15:55:36.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:36.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509513 2024-08-19T15:55:36.315 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509513 2024-08-19T15:55:36.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509513' 2024-08-19T15:55:36.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:55:36.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509513 -lt 64424509513 2024-08-19T15:55:36.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:55:36.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:55:36.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:55:36.827 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:55:36.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:55:36.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:55:36.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:55:36.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:55:36.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:55:37.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:55:37.111 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:55:37.111 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:55:37.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:55:37.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:37.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:55:37.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836559 2024-08-19T15:55:37.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836559 2024-08-19T15:55:37.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559' 2024-08-19T15:55:37.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:37.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:55:37.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673037 2024-08-19T15:55:37.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673037 2024-08-19T15:55:37.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673037' 2024-08-19T15:55:37.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:37.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:55:37.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509514 2024-08-19T15:55:37.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509514 2024-08-19T15:55:37.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673037 2-64424509514' 2024-08-19T15:55:37.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:37.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836559 2024-08-19T15:55:37.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:37.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:55:37.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836559 2024-08-19T15:55:37.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:37.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836559 2024-08-19T15:55:37.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836559' 2024-08-19T15:55:37.447 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836559 2024-08-19T15:55:37.448 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:37.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836558 -lt 21474836559 2024-08-19T15:55:37.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:38.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:55:38.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:38.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836558 -lt 21474836559 2024-08-19T15:55:38.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:39.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:55:39.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:40.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836559 -lt 21474836559 2024-08-19T15:55:40.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:40.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673037 2024-08-19T15:55:40.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:40.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:55:40.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673037 2024-08-19T15:55:40.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:40.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673037 2024-08-19T15:55:40.255 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673037 2024-08-19T15:55:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673037' 2024-08-19T15:55:40.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:55:40.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673037 -lt 42949673037 2024-08-19T15:55:40.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:40.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509514 2024-08-19T15:55:40.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:40.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:55:40.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509514 2024-08-19T15:55:40.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:40.521 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509514 2024-08-19T15:55:40.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509514 2024-08-19T15:55:40.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509514' 2024-08-19T15:55:40.522 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:55:40.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509515 -lt 64424509514 2024-08-19T15:55:40.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:55:40.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:55:40.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:55:41.065 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:55:41.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:55:41.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:55:41.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:55:41.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:55:41.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:55:41.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:55:41.355 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:55:41.355 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:55:41.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:55:41.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:41.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:55:41.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836561 2024-08-19T15:55:41.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836561 2024-08-19T15:55:41.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561' 2024-08-19T15:55:41.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:41.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:55:41.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673038 2024-08-19T15:55:41.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673038 2024-08-19T15:55:41.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673038' 2024-08-19T15:55:41.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:41.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:55:41.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509516 2024-08-19T15:55:41.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509516 2024-08-19T15:55:41.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673038 2-64424509516' 2024-08-19T15:55:41.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:41.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836561 2024-08-19T15:55:41.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:41.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:55:41.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836561 2024-08-19T15:55:41.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:41.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836561 2024-08-19T15:55:41.660 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836561 2024-08-19T15:55:41.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836561' 2024-08-19T15:55:41.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:41.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836560 -lt 21474836561 2024-08-19T15:55:41.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:42.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:55:42.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:43.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836561 -lt 21474836561 2024-08-19T15:55:43.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:43.220 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673038 2024-08-19T15:55:43.220 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:43.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:55:43.221 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673038 2024-08-19T15:55:43.222 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:43.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673038 2024-08-19T15:55:43.223 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673038 2024-08-19T15:55:43.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673038' 2024-08-19T15:55:43.223 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:55:43.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673039 -lt 42949673038 2024-08-19T15:55:43.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:43.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509516 2024-08-19T15:55:43.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:43.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:55:43.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509516 2024-08-19T15:55:43.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:43.503 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509516 2024-08-19T15:55:43.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509516 2024-08-19T15:55:43.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509516' 2024-08-19T15:55:43.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:55:43.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509517 -lt 64424509516 2024-08-19T15:55:43.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:55:43.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:55:43.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:55:44.013 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:55:44.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:55:44.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:55:44.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:55:44.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:55:44.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:55:44.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:55:44.287 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:55:44.287 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:55:44.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:55:44.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:44.287 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:55:44.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836562 2024-08-19T15:55:44.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836562 2024-08-19T15:55:44.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562' 2024-08-19T15:55:44.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:44.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:55:44.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673040 2024-08-19T15:55:44.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673040 2024-08-19T15:55:44.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673040' 2024-08-19T15:55:44.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:44.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:55:44.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509518 2024-08-19T15:55:44.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509518 2024-08-19T15:55:44.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673040 2-64424509518' 2024-08-19T15:55:44.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:44.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836562 2024-08-19T15:55:44.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:44.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:55:44.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836562 2024-08-19T15:55:44.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:44.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836562 2024-08-19T15:55:44.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836562' 2024-08-19T15:55:44.591 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836562 2024-08-19T15:55:44.591 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:44.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836561 -lt 21474836562 2024-08-19T15:55:44.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:45.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:55:45.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:46.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836562 -lt 21474836562 2024-08-19T15:55:46.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:46.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673040 2024-08-19T15:55:46.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:46.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:55:46.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673040 2024-08-19T15:55:46.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:46.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673040 2024-08-19T15:55:46.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673040' 2024-08-19T15:55:46.132 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673040 2024-08-19T15:55:46.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:55:46.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673040 -lt 42949673040 2024-08-19T15:55:46.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:46.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509518 2024-08-19T15:55:46.396 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:46.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:55:46.397 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509518 2024-08-19T15:55:46.397 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:46.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509518 2024-08-19T15:55:46.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509518' 2024-08-19T15:55:46.398 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509518 2024-08-19T15:55:46.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:55:46.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509518 -lt 64424509518 2024-08-19T15:55:46.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:55:46.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:55:46.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:55:46.909 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:55:46.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:55:46.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:55:46.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:55:46.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:55:46.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:55:47.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:55:47.183 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:55:47.183 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:55:47.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:55:47.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:47.184 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:55:47.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836564 2024-08-19T15:55:47.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836564 2024-08-19T15:55:47.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564' 2024-08-19T15:55:47.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:47.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:55:47.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673042 2024-08-19T15:55:47.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673042 2024-08-19T15:55:47.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673042' 2024-08-19T15:55:47.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:47.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:55:47.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509519 2024-08-19T15:55:47.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509519 2024-08-19T15:55:47.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673042 2-64424509519' 2024-08-19T15:55:47.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:47.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836564 2024-08-19T15:55:47.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:47.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:55:47.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836564 2024-08-19T15:55:47.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:47.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836564 2024-08-19T15:55:47.489 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836564 2024-08-19T15:55:47.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836564' 2024-08-19T15:55:47.490 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:47.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836563 -lt 21474836564 2024-08-19T15:55:47.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:48.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:55:48.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:49.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836563 -lt 21474836564 2024-08-19T15:55:49.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:50.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:55:50.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:50.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836564 -lt 21474836564 2024-08-19T15:55:50.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:50.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673042 2024-08-19T15:55:50.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:50.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:55:50.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673042 2024-08-19T15:55:50.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:50.311 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673042 2024-08-19T15:55:50.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673042 2024-08-19T15:55:50.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673042' 2024-08-19T15:55:50.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:55:50.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673042 -lt 42949673042 2024-08-19T15:55:50.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:50.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509519 2024-08-19T15:55:50.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:50.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:55:50.579 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509519 2024-08-19T15:55:50.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:50.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509519 2024-08-19T15:55:50.581 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509519 2024-08-19T15:55:50.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509519' 2024-08-19T15:55:50.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:55:50.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509520 -lt 64424509519 2024-08-19T15:55:50.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:55:50.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:55:50.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:55:51.095 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:55:51.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:55:51.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:55:51.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:55:51.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:55:51.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:55:51.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:55:51.368 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:55:51.368 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:55:51.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:55:51.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:51.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:55:51.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836566 2024-08-19T15:55:51.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836566 2024-08-19T15:55:51.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566' 2024-08-19T15:55:51.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:51.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:55:51.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673043 2024-08-19T15:55:51.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673043 2024-08-19T15:55:51.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673043' 2024-08-19T15:55:51.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:51.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:55:51.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509521 2024-08-19T15:55:51.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509521 2024-08-19T15:55:51.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673043 2-64424509521' 2024-08-19T15:55:51.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:51.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836566 2024-08-19T15:55:51.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:51.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:55:51.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836566 2024-08-19T15:55:51.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:51.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836566 2024-08-19T15:55:51.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836566' 2024-08-19T15:55:51.673 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836566 2024-08-19T15:55:51.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:51.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836565 -lt 21474836566 2024-08-19T15:55:51.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:52.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:55:52.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:53.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836566 -lt 21474836566 2024-08-19T15:55:53.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:53.222 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673043 2024-08-19T15:55:53.222 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:53.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:55:53.224 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673043 2024-08-19T15:55:53.224 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:53.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673043 2024-08-19T15:55:53.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-08-19T15:55:53.225 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673043 2024-08-19T15:55:53.225 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:55:53.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673044 -lt 42949673043 2024-08-19T15:55:53.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:53.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509521 2024-08-19T15:55:53.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:53.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:55:53.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509521 2024-08-19T15:55:53.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:53.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509521 2024-08-19T15:55:53.520 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509521 2024-08-19T15:55:53.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509521' 2024-08-19T15:55:53.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:55:53.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509522 -lt 64424509521 2024-08-19T15:55:53.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:55:53.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:55:53.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:55:54.040 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:55:54.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:55:54.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:55:54.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:55:54.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:55:54.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:55:54.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:55:54.313 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:55:54.313 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:55:54.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:55:54.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:54.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:55:54.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836567 2024-08-19T15:55:54.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836567 2024-08-19T15:55:54.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567' 2024-08-19T15:55:54.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:54.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:55:54.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673045 2024-08-19T15:55:54.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673045 2024-08-19T15:55:54.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673045' 2024-08-19T15:55:54.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:54.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:55:54.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509523 2024-08-19T15:55:54.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509523 2024-08-19T15:55:54.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673045 2-64424509523' 2024-08-19T15:55:54.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:54.616 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836567 2024-08-19T15:55:54.616 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:54.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:55:54.617 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836567 2024-08-19T15:55:54.617 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:54.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836567 2024-08-19T15:55:54.618 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836567 2024-08-19T15:55:54.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2024-08-19T15:55:54.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:54.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836566 -lt 21474836567 2024-08-19T15:55:54.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:55.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:55:55.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:56.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836567 -lt 21474836567 2024-08-19T15:55:56.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:56.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673045 2024-08-19T15:55:56.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:56.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:55:56.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673045 2024-08-19T15:55:56.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:56.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673045 2024-08-19T15:55:56.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673045' 2024-08-19T15:55:56.156 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673045 2024-08-19T15:55:56.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:55:56.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673045 -lt 42949673045 2024-08-19T15:55:56.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:56.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509523 2024-08-19T15:55:56.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:56.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:55:56.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509523 2024-08-19T15:55:56.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:56.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509523 2024-08-19T15:55:56.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509523' 2024-08-19T15:55:56.424 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509523 2024-08-19T15:55:56.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:55:56.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509523 -lt 64424509523 2024-08-19T15:55:56.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:55:56.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:55:56.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:55:56.980 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:55:56.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:55:56.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:55:56.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:55:56.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:55:56.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:55:57.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:55:57.257 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:55:57.257 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:55:57.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:55:57.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:57.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:55:57.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836569 2024-08-19T15:55:57.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836569 2024-08-19T15:55:57.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569' 2024-08-19T15:55:57.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:57.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:55:57.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673047 2024-08-19T15:55:57.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673047 2024-08-19T15:55:57.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673047' 2024-08-19T15:55:57.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:55:57.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:55:57.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509524 2024-08-19T15:55:57.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509524 2024-08-19T15:55:57.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673047 2-64424509524' 2024-08-19T15:55:57.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:57.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836569 2024-08-19T15:55:57.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:57.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:55:57.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836569 2024-08-19T15:55:57.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:57.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836569 2024-08-19T15:55:57.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-08-19T15:55:57.560 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836569 2024-08-19T15:55:57.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:57.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836568 -lt 21474836569 2024-08-19T15:55:57.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:55:58.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:55:58.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:55:59.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836569 -lt 21474836569 2024-08-19T15:55:59.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:59.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673047 2024-08-19T15:55:59.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:59.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:55:59.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673047 2024-08-19T15:55:59.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:59.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673047 2024-08-19T15:55:59.121 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673047 2024-08-19T15:55:59.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673047' 2024-08-19T15:55:59.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:55:59.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673047 -lt 42949673047 2024-08-19T15:55:59.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:55:59.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509524 2024-08-19T15:55:59.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:55:59.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:55:59.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509524 2024-08-19T15:55:59.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:55:59.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509524 2024-08-19T15:55:59.392 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509524 2024-08-19T15:55:59.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-08-19T15:55:59.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:55:59.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509525 -lt 64424509524 2024-08-19T15:55:59.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:55:59.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:55:59.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:55:59.913 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:55:59.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:55:59.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:55:59.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:55:59.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:55:59.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:56:00.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:56:00.188 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:56:00.188 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:56:00.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:56:00.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:00.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:56:00.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836570 2024-08-19T15:56:00.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836570 2024-08-19T15:56:00.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570' 2024-08-19T15:56:00.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:00.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:56:00.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673048 2024-08-19T15:56:00.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673048 2024-08-19T15:56:00.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673048' 2024-08-19T15:56:00.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:00.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:56:00.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509526 2024-08-19T15:56:00.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509526 2024-08-19T15:56:00.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673048 2-64424509526' 2024-08-19T15:56:00.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:00.536 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836570 2024-08-19T15:56:00.536 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:00.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:56:00.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836570 2024-08-19T15:56:00.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:00.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836570 2024-08-19T15:56:00.539 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836570 2024-08-19T15:56:00.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836570' 2024-08-19T15:56:00.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:56:00.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836569 -lt 21474836570 2024-08-19T15:56:00.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:56:01.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:56:01.802 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:56:02.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836571 -lt 21474836570 2024-08-19T15:56:02.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:02.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673048 2024-08-19T15:56:02.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:02.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:56:02.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673048 2024-08-19T15:56:02.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:02.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673048 2024-08-19T15:56:02.079 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673048 2024-08-19T15:56:02.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673048' 2024-08-19T15:56:02.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:56:02.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673048 -lt 42949673048 2024-08-19T15:56:02.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:02.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509526 2024-08-19T15:56:02.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:02.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:56:02.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509526 2024-08-19T15:56:02.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:02.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509526 2024-08-19T15:56:02.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509526' 2024-08-19T15:56:02.344 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509526 2024-08-19T15:56:02.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:56:02.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509526 -lt 64424509526 2024-08-19T15:56:02.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:56:02.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:56:02.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:56:02.858 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:56:02.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:56:02.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:56:02.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:56:02.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:56:02.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:56:03.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:56:03.132 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:56:03.132 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:56:03.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:56:03.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:03.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:56:03.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836572 2024-08-19T15:56:03.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836572 2024-08-19T15:56:03.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572' 2024-08-19T15:56:03.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:03.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:56:03.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673050 2024-08-19T15:56:03.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673050 2024-08-19T15:56:03.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673050' 2024-08-19T15:56:03.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:03.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:56:03.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509528 2024-08-19T15:56:03.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509528 2024-08-19T15:56:03.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673050 2-64424509528' 2024-08-19T15:56:03.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:03.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836572 2024-08-19T15:56:03.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:03.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:56:03.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836572 2024-08-19T15:56:03.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:03.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836572 2024-08-19T15:56:03.448 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836572 2024-08-19T15:56:03.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836572' 2024-08-19T15:56:03.449 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:56:03.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836571 -lt 21474836572 2024-08-19T15:56:03.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:56:04.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:56:04.709 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:56:04.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836571 -lt 21474836572 2024-08-19T15:56:05.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:56:05.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:56:05.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:56:06.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836572 -lt 21474836572 2024-08-19T15:56:06.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:06.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673050 2024-08-19T15:56:06.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:06.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:56:06.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673050 2024-08-19T15:56:06.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:06.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673050 2024-08-19T15:56:06.257 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673050 2024-08-19T15:56:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673050' 2024-08-19T15:56:06.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:56:06.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673050 -lt 42949673050 2024-08-19T15:56:06.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:06.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509528 2024-08-19T15:56:06.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:06.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:56:06.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509528 2024-08-19T15:56:06.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:06.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509528 2024-08-19T15:56:06.528 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509528 2024-08-19T15:56:06.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509528' 2024-08-19T15:56:06.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:56:06.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509528 -lt 64424509528 2024-08-19T15:56:06.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:56:06.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:56:06.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:56:07.044 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:56:07.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:56:07.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:56:07.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:56:07.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:56:07.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:56:07.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:56:07.344 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:56:07.344 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:56:07.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:56:07.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:07.344 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:56:07.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836574 2024-08-19T15:56:07.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836574 2024-08-19T15:56:07.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574' 2024-08-19T15:56:07.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:07.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:56:07.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673052 2024-08-19T15:56:07.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673052 2024-08-19T15:56:07.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673052' 2024-08-19T15:56:07.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:07.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:56:07.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509529 2024-08-19T15:56:07.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509529 2024-08-19T15:56:07.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673052 2-64424509529' 2024-08-19T15:56:07.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:07.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836574 2024-08-19T15:56:07.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:07.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:56:07.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836574 2024-08-19T15:56:07.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:07.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836574 2024-08-19T15:56:07.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836574' 2024-08-19T15:56:07.647 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836574 2024-08-19T15:56:07.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:56:07.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836573 -lt 21474836574 2024-08-19T15:56:07.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:56:08.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:56:08.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:56:09.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836574 -lt 21474836574 2024-08-19T15:56:09.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:09.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673052 2024-08-19T15:56:09.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:09.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:56:09.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673052 2024-08-19T15:56:09.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:09.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673052 2024-08-19T15:56:09.189 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673052 2024-08-19T15:56:09.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2024-08-19T15:56:09.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:56:09.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673052 -lt 42949673052 2024-08-19T15:56:09.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:09.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509529 2024-08-19T15:56:09.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:09.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:56:09.490 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509529 2024-08-19T15:56:09.490 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:09.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509529 2024-08-19T15:56:09.491 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509529 2024-08-19T15:56:09.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509529' 2024-08-19T15:56:09.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:56:09.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509530 -lt 64424509529 2024-08-19T15:56:09.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:56:09.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:56:09.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:56:10.003 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:56:10.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:56:10.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:56:10.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:56:10.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:56:10.014 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:56:10.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:56:10.275 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:56:10.275 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:56:10.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:56:10.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:10.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:56:10.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836575 2024-08-19T15:56:10.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836575 2024-08-19T15:56:10.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575' 2024-08-19T15:56:10.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:10.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:56:10.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673053 2024-08-19T15:56:10.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673053 2024-08-19T15:56:10.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673053' 2024-08-19T15:56:10.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:10.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:56:10.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509531 2024-08-19T15:56:10.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509531 2024-08-19T15:56:10.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673053 2-64424509531' 2024-08-19T15:56:10.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:10.576 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836575 2024-08-19T15:56:10.576 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:10.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:56:10.577 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836575 2024-08-19T15:56:10.577 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:10.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836575 2024-08-19T15:56:10.578 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836575 2024-08-19T15:56:10.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836575' 2024-08-19T15:56:10.579 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:56:10.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836574 -lt 21474836575 2024-08-19T15:56:10.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:56:11.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:56:11.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:56:12.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836576 -lt 21474836575 2024-08-19T15:56:12.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:12.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673053 2024-08-19T15:56:12.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:12.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:56:12.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673053 2024-08-19T15:56:12.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:12.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673053 2024-08-19T15:56:12.120 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673053 2024-08-19T15:56:12.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673053' 2024-08-19T15:56:12.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:56:12.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673053 -lt 42949673053 2024-08-19T15:56:12.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:12.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509531 2024-08-19T15:56:12.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:12.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:56:12.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509531 2024-08-19T15:56:12.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:12.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509531 2024-08-19T15:56:12.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509531' 2024-08-19T15:56:12.388 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509531 2024-08-19T15:56:12.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:56:12.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509531 -lt 64424509531 2024-08-19T15:56:12.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:56:12.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:56:12.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:56:12.903 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:56:12.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:56:12.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:56:12.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:56:12.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:56:12.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:56:13.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:56:13.178 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:56:13.178 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:56:13.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:56:13.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:13.179 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:56:13.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836577 2024-08-19T15:56:13.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836577 2024-08-19T15:56:13.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577' 2024-08-19T15:56:13.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:13.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:56:13.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673055 2024-08-19T15:56:13.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673055 2024-08-19T15:56:13.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673055' 2024-08-19T15:56:13.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:13.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:56:13.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509533 2024-08-19T15:56:13.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509533 2024-08-19T15:56:13.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673055 2-64424509533' 2024-08-19T15:56:13.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:13.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836577 2024-08-19T15:56:13.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:13.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:56:13.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836577 2024-08-19T15:56:13.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:13.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836577 2024-08-19T15:56:13.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2024-08-19T15:56:13.494 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836577 2024-08-19T15:56:13.494 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:56:13.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836576 -lt 21474836577 2024-08-19T15:56:13.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:56:14.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:56:14.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:56:15.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836576 -lt 21474836577 2024-08-19T15:56:15.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:56:16.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:56:16.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:56:16.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836577 -lt 21474836577 2024-08-19T15:56:16.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:16.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673055 2024-08-19T15:56:16.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:16.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:56:16.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673055 2024-08-19T15:56:16.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:16.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673055 2024-08-19T15:56:16.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-08-19T15:56:16.315 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673055 2024-08-19T15:56:16.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:56:16.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673055 -lt 42949673055 2024-08-19T15:56:16.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:16.579 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509533 2024-08-19T15:56:16.579 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:16.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:56:16.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509533 2024-08-19T15:56:16.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:16.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509533 2024-08-19T15:56:16.582 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509533 2024-08-19T15:56:16.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509533' 2024-08-19T15:56:16.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:56:16.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509533 -lt 64424509533 2024-08-19T15:56:16.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:56:16.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:56:16.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:56:17.094 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:56:17.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:56:17.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:56:17.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:56:17.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:56:17.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:56:17.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:56:17.372 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:56:17.372 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:56:17.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:56:17.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:17.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:56:17.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836579 2024-08-19T15:56:17.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836579 2024-08-19T15:56:17.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579' 2024-08-19T15:56:17.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:17.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:56:17.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673057 2024-08-19T15:56:17.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673057 2024-08-19T15:56:17.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673057' 2024-08-19T15:56:17.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:17.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:56:17.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509534 2024-08-19T15:56:17.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509534 2024-08-19T15:56:17.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673057 2-64424509534' 2024-08-19T15:56:17.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:17.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836579 2024-08-19T15:56:17.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:17.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:56:17.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836579 2024-08-19T15:56:17.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:17.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836579 2024-08-19T15:56:17.674 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836579 2024-08-19T15:56:17.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-08-19T15:56:17.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:56:17.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836578 -lt 21474836579 2024-08-19T15:56:17.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:56:18.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:56:18.939 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:56:19.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836579 -lt 21474836579 2024-08-19T15:56:19.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:19.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673057 2024-08-19T15:56:19.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:19.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:56:19.220 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673057 2024-08-19T15:56:19.221 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:19.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673057 2024-08-19T15:56:19.222 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673057 2024-08-19T15:56:19.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673057' 2024-08-19T15:56:19.222 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:56:19.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673057 -lt 42949673057 2024-08-19T15:56:19.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:19.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509534 2024-08-19T15:56:19.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:19.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:56:19.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509534 2024-08-19T15:56:19.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:19.513 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509534 2024-08-19T15:56:19.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509534 2024-08-19T15:56:19.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509534' 2024-08-19T15:56:19.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:56:19.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509535 -lt 64424509534 2024-08-19T15:56:19.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:56:19.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:56:19.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:56:20.024 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:56:20.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:56:20.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:56:20.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:56:20.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:56:20.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:56:20.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:56:20.298 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:56:20.298 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:56:20.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:56:20.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:20.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:56:20.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836580 2024-08-19T15:56:20.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836580 2024-08-19T15:56:20.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580' 2024-08-19T15:56:20.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:20.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:56:20.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673058 2024-08-19T15:56:20.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673058 2024-08-19T15:56:20.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673058' 2024-08-19T15:56:20.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:20.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:56:20.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509536 2024-08-19T15:56:20.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509536 2024-08-19T15:56:20.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673058 2-64424509536' 2024-08-19T15:56:20.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:20.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836580 2024-08-19T15:56:20.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:20.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:56:20.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836580 2024-08-19T15:56:20.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:20.602 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836580 2024-08-19T15:56:20.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836580 2024-08-19T15:56:20.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836580' 2024-08-19T15:56:20.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:56:20.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836579 -lt 21474836580 2024-08-19T15:56:20.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:56:21.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:56:21.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:56:22.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836581 -lt 21474836580 2024-08-19T15:56:22.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:22.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673058 2024-08-19T15:56:22.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:22.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:56:22.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673058 2024-08-19T15:56:22.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:22.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673058 2024-08-19T15:56:22.134 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673058 2024-08-19T15:56:22.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673058' 2024-08-19T15:56:22.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:56:22.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673058 -lt 42949673058 2024-08-19T15:56:22.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:22.398 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509536 2024-08-19T15:56:22.398 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:22.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:56:22.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509536 2024-08-19T15:56:22.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:22.401 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509536 2024-08-19T15:56:22.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509536 2024-08-19T15:56:22.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-08-19T15:56:22.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:56:22.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509536 -lt 64424509536 2024-08-19T15:56:22.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:56:22.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:56:22.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:56:22.912 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:56:22.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-08-19T15:56:22.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-08-19T15:56:22.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-08-19T15:56:22.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-19T15:56:22.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T15:56:23.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T15:56:23.189 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T15:56:23.189 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T15:56:23.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T15:56:23.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:23.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T15:56:23.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836582 2024-08-19T15:56:23.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836582 2024-08-19T15:56:23.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582' 2024-08-19T15:56:23.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:23.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T15:56:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673060 2024-08-19T15:56:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673060 2024-08-19T15:56:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673060' 2024-08-19T15:56:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T15:56:23.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T15:56:23.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509538 2024-08-19T15:56:23.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509538 2024-08-19T15:56:23.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673060 2-64424509538' 2024-08-19T15:56:23.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:23.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836582 2024-08-19T15:56:23.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:23.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T15:56:23.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836582 2024-08-19T15:56:23.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:23.494 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836582 2024-08-19T15:56:23.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836582 2024-08-19T15:56:23.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2024-08-19T15:56:23.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:56:23.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836581 -lt 21474836582 2024-08-19T15:56:23.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:56:24.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T15:56:24.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:56:25.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836581 -lt 21474836582 2024-08-19T15:56:25.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T15:56:26.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T15:56:26.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T15:56:26.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836582 -lt 21474836582 2024-08-19T15:56:26.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:26.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673060 2024-08-19T15:56:26.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:26.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T15:56:26.319 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673060 2024-08-19T15:56:26.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:26.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673060 2024-08-19T15:56:26.321 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673060 2024-08-19T15:56:26.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673060' 2024-08-19T15:56:26.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T15:56:26.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673060 -lt 42949673060 2024-08-19T15:56:26.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T15:56:26.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509538 2024-08-19T15:56:26.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T15:56:26.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T15:56:26.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509538 2024-08-19T15:56:26.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T15:56:26.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509538 2024-08-19T15:56:26.586 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509538 2024-08-19T15:56:26.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509538' 2024-08-19T15:56:26.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T15:56:26.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509538 -lt 64424509538 2024-08-19T15:56:26.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-08-19T15:56:26.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-08-19T15:56:26.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-08-19T15:56:27.096 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T15:56:27.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:383: _scrub_abort: break 2024-08-19T15:56:27.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:385: _scrub_abort: set +o pipefail 2024-08-19T15:56:27.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:387: _scrub_abort: sleep 5 2024-08-19T15:56:32.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:389: _scrub_abort: grep 'nodeep_scrub set, aborting' td/osd-scrub-test/osd.1.log 2024-08-19T15:56:32.123 INFO:tasks.workunit.client.0.smithi102.stdout:2024-08-19T15:54:28.877+0000 7fe6feb78640 10 osd.1 pg_epoch: 21 pg[1.0( v 19'1000 (0'0,19'1000] local-lis/les=17/18 n=1000 ec=17/17 lis/c=17/17 les/c/f=18/18/0 sis=17) [1,0,2] r=0 lpr=17 crt=19'1000 lcod 19'999 mlcod 19'999 active+clean+scrubbing+deep [ 1.0: ] TIME_FOR_DEEP] scrubber: nodeep_scrub set, aborting 2024-08-19T15:56:32.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:395: _scrub_abort: get_last_scrub_stamp 1.0 2024-08-19T15:56:32.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:32.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:32.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:32.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:32.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:395: _scrub_abort: local last_scrub=2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:32.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:396: _scrub_abort: ceph config set osd osd_scrub_sleep 0.1 2024-08-19T15:56:32.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:398: _scrub_abort: ceph osd unset nodeep-scrub 2024-08-19T15:56:33.116 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is unset 2024-08-19T15:56:33.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:399: _scrub_abort: '[' deep-scrub = deep-scrub ']' 2024-08-19T15:56:33.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:401: _scrub_abort: ceph osd unset noscrub 2024-08-19T15:56:33.493 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is unset 2024-08-19T15:56:33.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:403: _scrub_abort: TIMEOUT=500 2024-08-19T15:56:33.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:404: _scrub_abort: wait_for_scrub 1.0 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:33.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-08-19T15:56:33.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:33.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T15:56:33.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-08-19T15:56:33.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:33.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:33.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:33.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:33.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:33.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:33.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:33.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:34.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:34.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:34.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:34.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:34.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:34.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:34.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:35.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:35.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:36.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:36.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:36.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:36.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:36.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:36.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:36.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:36.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:36.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:37.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:37.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:37.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:37.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:37.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:37.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:37.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:37.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:37.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:38.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:38.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:38.617 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:38.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:38.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:38.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:38.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:38.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:38.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:39.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:39.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:39.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:39.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:39.885 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:39.885 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:39.885 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:40.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:40.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:41.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:41.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:41.151 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:41.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:41.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:41.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:41.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:41.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:41.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:42.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:42.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:42.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:42.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:42.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:42.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:42.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:42.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:42.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:43.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:43.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:43.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:43.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:43.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:43.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:43.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:43.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:43.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:44.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:44.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:44.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:44.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:44.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:44.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:44.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:45.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:45.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:46.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:46.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:46.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:46.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:46.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:46.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:46.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:46.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:46.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:47.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:47.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:47.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:47.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:47.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:47.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:47.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:47.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:47.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:48.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:48.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:48.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:48.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:48.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:48.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:48.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:49.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:49.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:50.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:50.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:50.047 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:50.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:50.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:50.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:50.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:50.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:50.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:51.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:51.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:51.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:51.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:51.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:51.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:51.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:51.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:51.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:52.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:52.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:52.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:52.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:52.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:52.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:52.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:52.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:52.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:53.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:53.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:53.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:53.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:53.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:53.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:53.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:54.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:54.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:55.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:55.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:55.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:55.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:55.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:55.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:55.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:55.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:55.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:56.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:56.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:56.448 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:56.448 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:56.449 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:56.449 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:56.449 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:56.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:56.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:57.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:57.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:57.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:57.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:57.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:57.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:57.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:57.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:57.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:56:58.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:56:58.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:56:58.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:56:58.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:56:58.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:56:58.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:56:58.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:56:59.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:56:59.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:00.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:00.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:00.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:00.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:00.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:00.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:00.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:00.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:00.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:01.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:01.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:01.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:01.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:01.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:01.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:01.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:01.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:01.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:02.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:02.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:02.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:02.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:02.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:02.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:02.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:03.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:03.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:04.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:04.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:04.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:04.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:04.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:04.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:04.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:04.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:04.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:05.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:05.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:05.360 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:05.360 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:05.360 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:05.360 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:05.360 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:05.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:05.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:06.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:06.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:06.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:06.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:06.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:06.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:06.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:06.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:06.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:07.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:07.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:07.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:07.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:07.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:07.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:07.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:08.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:08.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:09.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:09.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:09.166 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:09.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:09.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:09.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:09.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:09.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:09.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:10.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:10.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:10.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:10.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:10.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:10.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:10.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:10.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:10.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:11.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:11.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:11.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:11.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:11.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:11.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:11.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:11.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:11.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:12.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:12.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:12.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:12.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:12.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:12.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:12.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:13.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:13.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:14.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:14.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:14.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:14.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:14.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:14.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:14.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:15.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:15.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:15.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:15.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:15.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:15.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:15.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:15.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:15.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:16.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:16.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:16.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:16.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:16.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:16.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:16.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:17.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:17.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:18.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:18.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:18.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:18.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:18.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:18.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:18.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:18.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:18.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:19.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:19.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:19.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:19.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:19.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:19.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:19.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:19.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:19.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:20.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:20.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:20.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:20.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:20.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:20.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:20.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:20.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:20.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:21.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:21.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:21.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:21.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:21.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:21.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:21.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:22.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:22.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:23.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:23.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:23.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:23.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:23.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:23.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:23.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:23.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:23.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:24.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:24.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:24.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:24.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:24.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:24.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:24.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:24.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:24.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:25.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:25.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:25.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:25.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:25.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:25.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:25.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:25.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:25.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:26.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:26.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:26.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:26.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:26.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:26.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:26.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:27.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:27.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:28.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:28.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:28.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:28.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:28.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:28.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:28.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:28.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:28.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:29.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:29.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:29.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:29.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:29.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:29.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:29.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:29.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:29.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:30.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:30.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:30.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:30.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:30.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:30.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:30.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:31.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:31.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:32.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:32.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:32.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:32.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:32.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:32.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:32.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:32.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:32.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:33.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:33.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:33.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:33.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:33.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:33.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:33.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:33.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:33.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:34.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:34.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:34.624 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:34.624 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:34.624 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:34.624 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:34.624 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:34.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:34.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:35.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:35.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:35.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:35.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:35.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:35.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:35.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:36.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:36.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:37.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:37.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:37.207 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:37.207 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:37.207 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:37.208 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:37.208 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:37.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:37.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:38.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:38.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:38.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:38.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:38.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:38.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:38.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:38.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:38.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:39.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:39.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:39.693 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:39.693 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:39.693 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:39.693 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:39.693 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:39.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:39.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:40.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:40.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:40.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:40.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:40.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:40.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:40.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:41.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:41.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:42.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:42.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:42.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:42.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:42.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:42.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:42.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:42.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:42.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:43.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:43.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:43.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:43.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:43.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:43.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:43.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:43.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:43.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:44.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:44.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:44.780 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:44.780 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:44.780 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:44.780 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:44.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:45.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:45.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:46.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:46.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:46.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:46.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:46.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:46.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:46.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:46.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:46.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:47.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:47.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:47.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:47.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:47.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:47.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:47.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:47.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:47.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:48.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:48.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:48.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:48.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:48.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:48.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:48.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:48.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:48.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:49.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:49.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:49.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:49.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:49.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:49.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:49.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:50.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:50.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:51.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:51.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:51.165 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:51.165 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:51.165 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:51.165 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:51.165 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:51.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:51.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:52.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:52.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:52.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:52.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:52.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:52.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:52.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:52.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:52.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:53.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:53.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:53.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:53.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:53.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:53.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:53.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:53.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:53.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:54.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:54.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:54.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:54.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:54.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:54.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:54.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:55.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:55.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:56.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:56.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:56.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:56.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:56.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:56.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:56.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:56.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:56.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:57.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:57.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:57.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:57.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:57.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:57.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:57.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:57.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:57.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:57:58.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:57:58.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:57:58.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:57:58.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:57:58.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:57:58.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:57:58.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:57:59.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:57:59.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:00.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:00.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:00.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:00.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:00.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:00.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:00.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:00.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:00.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:01.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:01.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:01.356 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:01.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:01.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:01.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:01.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:01.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:01.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:02.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:02.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:02.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:02.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:02.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:02.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:02.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:02.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:02.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:03.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:03.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:03.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:03.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:03.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:03.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:03.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:04.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:04.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:05.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:05.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:05.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:05.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:05.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:05.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:05.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:05.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:05.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:06.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:06.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:06.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:06.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:06.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:06.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:06.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:06.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:06.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:07.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:07.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:07.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:07.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:07.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:07.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:07.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:07.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:07.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:08.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:08.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:08.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:08.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:08.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:08.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:08.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:09.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:09.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:10.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:10.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:10.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:10.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:10.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:10.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:10.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:10.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:10.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:11.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:11.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:11.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:11.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:11.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:11.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:11.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:11.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:11.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:12.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:12.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:12.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:12.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:12.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:12.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:12.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:13.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:13.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:14.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:14.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:14.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:14.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:14.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:14.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:14.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:14.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:14.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:15.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:15.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:15.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:15.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:15.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:15.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:15.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:15.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:15.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:16.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:16.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:16.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:16.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:16.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:16.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:16.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:16.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:16.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:17.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:17.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:17.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:17.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:17.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:17.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:17.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:18.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:18.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:19.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:19.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:19.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:19.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:19.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:19.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:19.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:19.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:19.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:20.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:20.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:20.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:20.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:20.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:20.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:20.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:20.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:20.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:21.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:21.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:21.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:21.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:21.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:21.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:21.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:21.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:21.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:22.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:22.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:22.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:22.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:22.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:22.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:22.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:23.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:23.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:24.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:24.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:24.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:24.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:24.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:24.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:24.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:24.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:24.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:25.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:25.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:25.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:25.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:25.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:25.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:25.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:25.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:25.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:26.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:26.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:26.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:26.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:26.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:26.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:26.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:27.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:27.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:28.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:28.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:28.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:28.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:28.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:28.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:28.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:28.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:28.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:29.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:29.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:29.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:29.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:29.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:29.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:29.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:29.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:29.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:30.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:30.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:30.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:30.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:30.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:30.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:30.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:30.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:30.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:31.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:31.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:31.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:31.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:31.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:31.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:31.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:32.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:32.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:33.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:33.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:33.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:33.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:33.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:33.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:33.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:33.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:33.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:34.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:34.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:34.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:34.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:34.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:34.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:34.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:34.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:34.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:35.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:35.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:35.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:35.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:35.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:35.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:35.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:35.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:35.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:36.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:36.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:36.979 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:36.979 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:36.979 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:36.979 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:36.979 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:37.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:37.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:38.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:38.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:38.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:38.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:38.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:38.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:38.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:38.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:38.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:39.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:39.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:39.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:39.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:39.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:39.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:39.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:39.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:39.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:40.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:40.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:40.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:40.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:40.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:40.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:40.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:41.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:41.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:42.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:42.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:42.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:42.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:42.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:42.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:42.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:42.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:42.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:43.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:43.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:43.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:43.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:43.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:43.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:43.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:43.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:43.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:44.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:44.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:44.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:44.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:44.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:44.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:44.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:44.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:44.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:45.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:45.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:45.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:45.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:45.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:45.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:45.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:46.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:46.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:47.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:47.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:47.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:47.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:47.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:47.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:47.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:47.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:47.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:48.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:48.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:48.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:48.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:48.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:48.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:48.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:48.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:48.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:49.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:49.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:49.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:49.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:49.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:49.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:49.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:49.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:49.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:50.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:50.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:50.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:50.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:50.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:50.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:50.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:51.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:51.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:52.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:52.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:52.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:52.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:52.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:52.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:52.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:52.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:52.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:53.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:53.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:53.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:53.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:53.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:53.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:53.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:53.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:53.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:54.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:54.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:54.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:54.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:54.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:54.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:54.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:55.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:55.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:56.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:56.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:56.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:56.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:56.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:56.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:56.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:56.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:56.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:57.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:57.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:57.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:57.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:57.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:57.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:57.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:57.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:57.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:58.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:58.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:58.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:58.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:58.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:58.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:58.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:58:58.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:58:58.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:58:59.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:58:59.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:58:59.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:58:59.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:58:59.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:58:59.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:58:59.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:00.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:00.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:01.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:01.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:01.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:01.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:01.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:01.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:01.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:01.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:01.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:02.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:02.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:02.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:02.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:02.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:02.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:02.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:02.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:02.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:03.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:03.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:03.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:03.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:03.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:03.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:03.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:03.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:03.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:04.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:04.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:04.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:04.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:04.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:04.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:04.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:05.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:05.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:06.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:06.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:06.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:06.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:06.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:06.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:06.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:06.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:06.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:07.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:07.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:07.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:07.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:07.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:07.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:07.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:07.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:07.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:08.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:08.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:08.806 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:08.806 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:08.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:08.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:08.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:09.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:09.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:10.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:10.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:10.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:10.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:10.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:10.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:10.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:10.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:10.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:11.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:11.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:11.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:11.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:11.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:11.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:11.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:11.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:11.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:12.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:12.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:12.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:12.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:12.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:12.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:12.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:12.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:12.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:13.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:13.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:13.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:13.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:13.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:13.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:13.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:14.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:14.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:15.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:15.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:15.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:15.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:15.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:15.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:15.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:15.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:15.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:16.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:16.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:16.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:16.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:16.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:16.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:16.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:16.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:16.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:17.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:17.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:17.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:17.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:17.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:17.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:17.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:17.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:17.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:18.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:18.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:18.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:18.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:18.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:18.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:18.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:19.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:19.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:20.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:20.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:20.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:20.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:20.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:20.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:20.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:20.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:20.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:21.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:21.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:21.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:21.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:21.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:21.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:21.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:21.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:21.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:22.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:22.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:22.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:22.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:22.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:22.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:22.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:23.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:23.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:24.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:24.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:24.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:24.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:24.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:24.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:24.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:24.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:24.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:25.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:25.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:25.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:25.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:25.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:25.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:25.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:25.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:25.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:26.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:26.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:26.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:26.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:26.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:26.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:26.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:26.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:26.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:27.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:27.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:27.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:27.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:27.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:27.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:27.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:28.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:28.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:29.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:29.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:29.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:29.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:29.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:29.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:29.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:29.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:29.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:30.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:30.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:30.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:30.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:30.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:30.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:30.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:30.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:30.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:31.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:31.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:31.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:31.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:31.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:31.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:31.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:31.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:31.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:32.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:32.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:32.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:32.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:32.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:32.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:32.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:33.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:33.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:34.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:34.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:34.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:34.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:34.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:34.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:34.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:35.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:35.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:35.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:35.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:35.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:35.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:35.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:35.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:35.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:36.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:36.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:36.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:36.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:36.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:36.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:36.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:37.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:37.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:38.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:38.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:38.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:38.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:38.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:38.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:38.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:38.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:38.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:39.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:39.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:39.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:39.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:39.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:39.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:39.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:39.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:39.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:40.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:40.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:40.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:40.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:40.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:40.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:40.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:40.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:40.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:41.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:41.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:41.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:41.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:41.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:41.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:41.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:42.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:42.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:43.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:43.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:43.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:43.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:43.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:43.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:43.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:43.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:43.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:44.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:44.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:44.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:44.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:44.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:44.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:44.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:44.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:44.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:45.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:45.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:45.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:45.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:45.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:45.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:45.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:45.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:45.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:46.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:46.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:46.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:46.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:46.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:46.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:46.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:47.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:47.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:48.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:48.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:48.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:48.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:48.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:48.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:48.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:48.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:48.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:49.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:49.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:49.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:49.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:49.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:49.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:49.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:49.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:49.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:50.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:50.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:50.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:50.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:50.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:50.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:50.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:51.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:51.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:52.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:52.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:52.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:52.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:52.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:52.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:52.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:52.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:52.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:53.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:53.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:53.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:53.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:53.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:53.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:53.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:53.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:53.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:54.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:54.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:54.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:54.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:54.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:54.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:54.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:54.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:54.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:55.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:55.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:55.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:55.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:55.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:55.922 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:55.922 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:56.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:56.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:57.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:57.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:57.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:57.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:57.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:57.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:57.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:57.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:57.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:58.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:58.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:58.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:58.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:58.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:58.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:58.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:58.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:58.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T15:59:59.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T15:59:59.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T15:59:59.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T15:59:59.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T15:59:59.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T15:59:59.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T15:59:59.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T15:59:59.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T15:59:59.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T16:00:00.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T16:00:00.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T16:00:00.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T16:00:00.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T16:00:00.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T16:00:00.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T16:00:00.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T16:00:01.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T16:00:01.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T16:00:02.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T16:00:02.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T16:00:02.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T16:00:02.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T16:00:02.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T16:00:02.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T16:00:02.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T16:00:02.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T16:00:02.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T16:00:03.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T16:00:03.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T16:00:03.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T16:00:03.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T16:00:03.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T16:00:03.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T16:00:03.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T16:00:03.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T16:00:03.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T16:00:04.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T16:00:04.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T16:00:04.820 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T16:00:04.820 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T16:00:04.820 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T16:00:04.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T16:00:04.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T16:00:05.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T16:00:05.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T16:00:06.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T16:00:06.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T16:00:06.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T16:00:06.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T16:00:06.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T16:00:06.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T16:00:06.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T16:00:06.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-05T15:52:33.397590+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T16:00:06.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-08-19T16:00:07.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-08-19T16:00:07.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-08-19T16:00:07.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T16:00:07.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T16:00:07.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T16:00:07.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T16:00:07.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T16:00:07.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-08-19T16:00:01.813077+0000 '>' 2024-08-05T15:52:33.397590+0000 2024-08-19T16:00:07.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-08-19T16:00:07.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:405: _scrub_abort: perf_counters td/osd-scrub-test 3 2024-08-19T16:00:07.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2024-08-19T16:00:07.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2024-08-19T16:00:07.650 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2024-08-19T16:00:07.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2024-08-19T16:00:07.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T16:00:07.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2024-08-19T16:00:07.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-08-19T16:00:07.745 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T16:00:07.745 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_dp_ec": [ 2024-08-19T16:00:07.745 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:00:07.745 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:00:07.745 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "deep", 2024-08-19T16:00:07.745 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "ec" 2024-08-19T16:00:07.745 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.746 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:00:07.746 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:00:07.746 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:00:07.746 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:00:07.746 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:00:07.746 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.746 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.746 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.746 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.746 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:00:07.746 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:00:07.746 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.746 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.747 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.747 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.747 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:00:07.747 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:00:07.747 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:00:07.747 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:00:07.747 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:00:07.747 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:00:07.747 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:00:07.747 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.747 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.747 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.748 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.748 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:00:07.748 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:00:07.748 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:00:07.748 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:00:07.748 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.748 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.748 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.748 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.748 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:00:07.748 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.748 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.749 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:00:07.749 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_dp_repl": [ 2024-08-19T16:00:07.749 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:00:07.749 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:00:07.749 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "deep", 2024-08-19T16:00:07.749 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "replicated" 2024-08-19T16:00:07.749 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.749 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:00:07.749 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:00:07.749 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:00:07.749 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:00:07.749 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:00:07.750 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.750 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.750 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.750 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.750 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:00:07.750 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:00:07.750 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.750 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.750 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.750 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.750 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:00:07.750 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:00:07.751 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:00:07.751 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:00:07.751 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:00:07.751 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:00:07.751 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:00:07.751 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.751 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.751 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.751 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.751 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:00:07.751 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:00:07.751 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:00:07.751 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:00:07.752 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.752 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.752 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.752 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.752 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:00:07.752 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.752 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.752 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:00:07.752 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_sh_ec": [ 2024-08-19T16:00:07.752 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:00:07.752 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:00:07.752 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "shallow", 2024-08-19T16:00:07.752 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "ec" 2024-08-19T16:00:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:00:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:00:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:00:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:00:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:00:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:00:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:00:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.754 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.754 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.754 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.754 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:00:07.754 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:00:07.754 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:00:07.754 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:00:07.754 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:00:07.754 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:00:07.754 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:00:07.754 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.754 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.754 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.755 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.755 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:00:07.755 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:00:07.755 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:00:07.755 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:00:07.755 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.755 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.755 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.755 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.755 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:00:07.755 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.755 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.755 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:00:07.756 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_sh_repl": [ 2024-08-19T16:00:07.756 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:00:07.756 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:00:07.756 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "shallow", 2024-08-19T16:00:07.756 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "replicated" 2024-08-19T16:00:07.756 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.756 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:00:07.756 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:00:07.756 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:00:07.756 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:00:07.756 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:00:07.756 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.757 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.757 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.757 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.757 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:00:07.757 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:00:07.758 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.758 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.758 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.758 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.758 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:00:07.758 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:00:07.758 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:00:07.758 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:00:07.758 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:00:07.759 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:00:07.759 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:00:07.759 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.759 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.759 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.759 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.759 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:00:07.759 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:00:07.759 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:00:07.759 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:00:07.759 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.760 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.760 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.760 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.760 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:00:07.760 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.760 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.760 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T16:00:07.760 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:00:07.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T16:00:07.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2024-08-19T16:00:07.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-08-19T16:00:07.845 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T16:00:07.846 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_dp_ec": [ 2024-08-19T16:00:07.846 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:00:07.846 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:00:07.846 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "deep", 2024-08-19T16:00:07.846 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "ec" 2024-08-19T16:00:07.846 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.846 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:00:07.846 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:00:07.846 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:00:07.846 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:00:07.846 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:00:07.846 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.847 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.847 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.847 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.847 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:00:07.847 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:00:07.847 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.847 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.847 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.847 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.847 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:00:07.847 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:00:07.847 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:00:07.848 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:00:07.848 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:00:07.848 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:00:07.848 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:00:07.848 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.848 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.848 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.848 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.848 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:00:07.848 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:00:07.848 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:00:07.849 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:00:07.849 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.849 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.849 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.849 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.849 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:00:07.850 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.850 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.850 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:00:07.850 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_dp_repl": [ 2024-08-19T16:00:07.850 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:00:07.850 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:00:07.850 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "deep", 2024-08-19T16:00:07.850 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "replicated" 2024-08-19T16:00:07.850 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.851 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:00:07.851 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 2, 2024-08-19T16:00:07.851 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 2, 2024-08-19T16:00:07.851 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 1, 2024-08-19T16:00:07.851 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:00:07.851 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 1, 2024-08-19T16:00:07.851 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 206.907756402, 2024-08-19T16:00:07.851 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 206.907756402 2024-08-19T16:00:07.851 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.851 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 1, 2024-08-19T16:00:07.851 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:00:07.851 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 1, 2024-08-19T16:00:07.851 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 10.257789538, 2024-08-19T16:00:07.852 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 10.257789538 2024-08-19T16:00:07.852 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.852 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:00:07.852 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 42, 2024-08-19T16:00:07.852 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:00:07.852 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:00:07.852 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:00:07.853 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 2, 2024-08-19T16:00:07.853 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:00:07.853 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 2, 2024-08-19T16:00:07.853 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0.002999938, 2024-08-19T16:00:07.853 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0.001499969 2024-08-19T16:00:07.853 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.854 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:00:07.854 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:00:07.854 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:00:07.854 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:00:07.854 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.855 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.855 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.855 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.855 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 2 2024-08-19T16:00:07.855 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.855 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.856 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:00:07.856 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_sh_ec": [ 2024-08-19T16:00:07.856 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:00:07.856 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:00:07.856 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "shallow", 2024-08-19T16:00:07.857 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "ec" 2024-08-19T16:00:07.857 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.857 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:00:07.857 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:00:07.858 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:00:07.858 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:00:07.858 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:00:07.858 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.858 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.859 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.859 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.859 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:00:07.859 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:00:07.859 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.859 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.860 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.860 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.860 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:00:07.860 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:00:07.860 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:00:07.860 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:00:07.860 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:00:07.861 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:00:07.861 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:00:07.861 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.861 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.861 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.861 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.861 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:00:07.861 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:00:07.861 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:00:07.861 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:00:07.861 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.861 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.861 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.862 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.862 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:00:07.862 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.862 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.862 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:00:07.862 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_sh_repl": [ 2024-08-19T16:00:07.862 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:00:07.862 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:00:07.862 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "shallow", 2024-08-19T16:00:07.862 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "replicated" 2024-08-19T16:00:07.863 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.863 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:00:07.863 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:00:07.863 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:00:07.863 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:00:07.863 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:00:07.864 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.864 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.864 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.864 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.864 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:00:07.864 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:00:07.864 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.864 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.864 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.864 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.864 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:00:07.865 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:00:07.865 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:00:07.865 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:00:07.865 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:00:07.865 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:00:07.865 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:00:07.865 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.865 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.866 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.866 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.866 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:00:07.866 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:00:07.866 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:00:07.866 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:00:07.866 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.866 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.866 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.866 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.866 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:00:07.866 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.866 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.867 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T16:00:07.867 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:00:07.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T16:00:07.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2024-08-19T16:00:07.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-08-19T16:00:07.945 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T16:00:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_dp_ec": [ 2024-08-19T16:00:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:00:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:00:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "deep", 2024-08-19T16:00:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "ec" 2024-08-19T16:00:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:00:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:00:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:00:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:00:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:00:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:00:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:00:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:00:07.948 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:00:07.948 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:00:07.948 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:00:07.948 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:00:07.948 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:00:07.948 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:00:07.948 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.948 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.948 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.948 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.948 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:00:07.949 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:00:07.949 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:00:07.949 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:00:07.949 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.949 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.949 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.949 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.949 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:00:07.950 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.950 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.950 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:00:07.950 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_dp_repl": [ 2024-08-19T16:00:07.950 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:00:07.950 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:00:07.950 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "deep", 2024-08-19T16:00:07.950 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "replicated" 2024-08-19T16:00:07.950 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.950 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:00:07.950 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:00:07.950 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:00:07.951 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:00:07.951 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:00:07.951 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.951 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.951 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.951 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.951 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:00:07.951 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:00:07.951 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.951 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.951 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.951 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.951 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:00:07.951 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:00:07.952 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:00:07.952 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:00:07.952 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:00:07.952 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:00:07.952 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:00:07.952 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.952 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.952 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.952 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.952 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:00:07.953 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:00:07.953 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:00:07.953 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:00:07.953 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.953 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.953 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.953 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.953 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:00:07.953 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.953 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.953 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:00:07.954 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_sh_ec": [ 2024-08-19T16:00:07.954 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:00:07.954 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:00:07.954 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "shallow", 2024-08-19T16:00:07.954 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "ec" 2024-08-19T16:00:07.954 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.954 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:00:07.954 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:00:07.954 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:00:07.954 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:00:07.954 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:00:07.954 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.954 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.955 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.955 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.955 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:00:07.955 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:00:07.955 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.955 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.955 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.955 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.955 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:00:07.955 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:00:07.955 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:00:07.955 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:00:07.955 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:00:07.956 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:00:07.956 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:00:07.956 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.956 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.956 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.956 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.956 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:00:07.956 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:00:07.956 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:00:07.956 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:00:07.956 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.956 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.956 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.956 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.957 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:00:07.957 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.957 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.957 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:00:07.957 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_sh_repl": [ 2024-08-19T16:00:07.957 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:00:07.957 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:00:07.957 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "shallow", 2024-08-19T16:00:07.957 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "replicated" 2024-08-19T16:00:07.957 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.957 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:00:07.957 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:00:07.957 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:00:07.958 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:00:07.958 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:00:07.958 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.958 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.958 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.958 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.958 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:00:07.958 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:00:07.958 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.958 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.958 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.959 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.959 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:00:07.959 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:00:07.959 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:00:07.959 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:00:07.959 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:00:07.959 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:00:07.960 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:00:07.960 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.960 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.960 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.960 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.960 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:00:07.960 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:00:07.960 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:00:07.960 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:00:07.960 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:00:07.960 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:00:07.960 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:00:07.961 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:00:07.961 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:00:07.961 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.961 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:00:07.961 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T16:00:07.961 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:00:07.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_deep_scrub_abort ------------------' 2024-08-19T16:00:07.962 INFO:tasks.workunit.client.0.smithi102.stdout:-------------- Teardown Test TEST_deep_scrub_abort ------------------ 2024-08-19T16:00:07.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2024-08-19T16:00:07.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-08-19T16:00:07.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T16:00:07.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-08-19T16:00:07.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T16:00:07.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T16:00:07.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T16:00:07.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T16:00:07.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T16:00:08.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T16:00:08.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T16:00:08.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T16:00:08.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T16:00:08.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T16:00:08.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T16:00:08.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T16:00:08.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T16:00:08.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T16:00:08.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T16:00:08.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T16:00:08.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T16:00:08.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T16:00:08.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-08-19T16:00:08.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T16:00:08.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:00:08.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:00:08.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.536210 2024-08-19T16:00:08.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T16:00:08.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T16:00:08.121 INFO:tasks.workunit.client.0.smithi102.stdout:-------------- Complete Test TEST_deep_scrub_abort ------------------ 2024-08-19T16:00:08.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_deep_scrub_abort ------------------' 2024-08-19T16:00:08.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-08-19T16:00:08.122 INFO:tasks.workunit.client.0.smithi102.stdout:-------------- Prepare Test TEST_dump_scrub_schedule ------------------- 2024-08-19T16:00:08.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_dump_scrub_schedule -------------------' 2024-08-19T16:00:08.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-08-19T16:00:08.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-08-19T16:00:08.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-08-19T16:00:08.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-08-19T16:00:08.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T16:00:08.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-08-19T16:00:08.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T16:00:08.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T16:00:08.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T16:00:08.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T16:00:08.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T16:00:08.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T16:00:08.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T16:00:08.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T16:00:08.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T16:00:08.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T16:00:08.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T16:00:08.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T16:00:08.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T16:00:08.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T16:00:08.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T16:00:08.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T16:00:08.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T16:00:08.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T16:00:08.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-08-19T16:00:08.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T16:00:08.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:00:08.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:00:08.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.536210 2024-08-19T16:00:08.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T16:00:08.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T16:00:08.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-08-19T16:00:08.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T16:00:08.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:00:08.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:00:08.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.536210 2024-08-19T16:00:08.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T16:00:08.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T16:00:08.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T16:00:08.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-08-19T16:00:08.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_dump_scrub_schedule -----------------------' 2024-08-19T16:00:08.140 INFO:tasks.workunit.client.0.smithi102.stdout:-------------- Run Test TEST_dump_scrub_schedule ----------------------- 2024-08-19T16:00:08.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_dump_scrub_schedule td/osd-scrub-test 2024-08-19T16:00:08.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:526: TEST_dump_scrub_schedule: local dir=td/osd-scrub-test 2024-08-19T16:00:08.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:527: TEST_dump_scrub_schedule: local poolname=test 2024-08-19T16:00:08.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:528: TEST_dump_scrub_schedule: local OSDS=3 2024-08-19T16:00:08.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:529: TEST_dump_scrub_schedule: local objects=15 2024-08-19T16:00:08.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:531: TEST_dump_scrub_schedule: TESTDATA=testdata.536210 2024-08-19T16:00:08.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:533: TEST_dump_scrub_schedule: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-08-19T16:00:08.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-test 2024-08-19T16:00:08.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T16:00:08.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T16:00:08.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T16:00:08.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-test/a 2024-08-19T16:00:08.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-08-19T16:00:08.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T16:00:08.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T16:00:08.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T16:00:08.509 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T16:00:08.509 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:00:08.509 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:00:08.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T16:00:08.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-08-19T16:00:08.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T16:00:08.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T16:00:08.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T16:00:08.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T16:00:08.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T16:00:08.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T16:00:08.778 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T16:00:08.778 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T16:00:08.779 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T16:00:08.779 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T16:00:08.779 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:00:08.779 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:00:08.779 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.536210/ceph-mon.a.asok 2024-08-19T16:00:08.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T16:00:08.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.536210/ceph-mon.a.asok config get fsid 2024-08-19T16:00:08.858 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T16:00:08.858 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T16:00:08.858 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T16:00:08.858 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T16:00:08.858 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T16:00:08.858 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T16:00:08.858 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T16:00:08.858 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T16:00:08.858 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T16:00:08.859 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:00:08.859 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:00:08.859 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.536210/ceph-mon.a.asok 2024-08-19T16:00:08.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T16:00:08.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.536210/ceph-mon.a.asok config get mon_host 2024-08-19T16:00:08.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:534: TEST_dump_scrub_schedule: run_mgr td/osd-scrub-test x 2024-08-19T16:00:08.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-test 2024-08-19T16:00:08.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T16:00:08.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T16:00:08.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T16:00:08.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-test/x 2024-08-19T16:00:08.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T16:00:09.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T16:00:09.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T16:00:09.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T16:00:09.737 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T16:00:09.737 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:00:09.737 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:00:09.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T16:00:09.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T16:00:09.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T16:00:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:545: TEST_dump_scrub_schedule: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2' 2024-08-19T16:00:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:547: TEST_dump_scrub_schedule: expr 3 - 1 2024-08-19T16:00:09.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:547: TEST_dump_scrub_schedule: seq 0 2 2024-08-19T16:00:09.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:547: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T16:00:09.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 0 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 2024-08-19T16:00:09.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-08-19T16:00:09.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T16:00:09.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T16:00:09.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T16:00:09.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/0 2024-08-19T16:00:09.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-08-19T16:00:09.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T16:00:09.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T16:00:09.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T16:00:09.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-08-19T16:00:09.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-08-19T16:00:09.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T16:00:09.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T16:00:09.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-08-19T16:00:09.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T16:00:09.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T16:00:09.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T16:00:09.770 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T16:00:09.770 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:00:09.770 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:00:09.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T16:00:09.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T16:00:09.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T16:00:09.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T16:00:09.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T16:00:09.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-08-19T16:00:09.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-08-19T16:00:09.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T16:00:09.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T16:00:09.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T16:00:09.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T16:00:09.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T16:00:09.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2' 2024-08-19T16:00:09.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/0 2024-08-19T16:00:09.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T16:00:09.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=138b2521-7ff2-424c-abe4-d626ef1b2b9c 2024-08-19T16:00:09.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 138b2521-7ff2-424c-abe4-d626ef1b2b9c' 2024-08-19T16:00:09.772 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 138b2521-7ff2-424c-abe4-d626ef1b2b9c 2024-08-19T16:00:09.772 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T16:00:09.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAJbMNmxYrDLhAAmcnl1Zih7qtDSLBotVebPg== 2024-08-19T16:00:09.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAJbMNmxYrDLhAAmcnl1Zih7qtDSLBotVebPg=="}' 2024-08-19T16:00:09.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 138b2521-7ff2-424c-abe4-d626ef1b2b9c -i td/osd-scrub-test/0/new.json 2024-08-19T16:00:09.985 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T16:00:09.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/0/new.json 2024-08-19T16:00:10.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 --mkfs --key AQAJbMNmxYrDLhAAmcnl1Zih7qtDSLBotVebPg== --osd-uuid 138b2521-7ff2-424c-abe4-d626ef1b2b9c 2024-08-19T16:00:10.030 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:10.029+0000 7f6109a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:00:10.035 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:10.034+0000 7f6109a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:00:10.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:10.035+0000 7f6109a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:00:10.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:10.036+0000 7f6109a0d640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-08-19T16:00:10.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:10.036+0000 7f6109a0d640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-08-19T16:00:10.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:10.036+0000 7f6109a0d640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-08-19T16:00:10.136 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:10.134+0000 7f6109a0d640 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-08-19T16:00:15.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-08-19T16:00:15.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T16:00:15.627 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T16:00:15.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T16:00:15.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T16:00:15.972 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T16:00:15.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T16:00:15.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 2024-08-19T16:00:15.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T16:00:15.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T16:00:15.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T16:00:16.001 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:16.000+0000 7f16f082e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:00:16.003 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:16.002+0000 7f16f082e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:00:16.004 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:16.003+0000 7f16f082e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:00:16.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T16:00:16.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T16:00:16.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T16:00:16.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T16:00:16.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T16:00:16.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:00:16.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T16:00:16.237 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T16:00:16.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:00:16.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:00:16.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:00:16.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:16.531+0000 7f16f082e640 -1 Falling back to public interface 2024-08-19T16:00:17.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:00:17.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:00:17.501 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T16:00:17.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T16:00:17.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:00:17.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:00:17.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:00:18.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:00:18.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:00:18.776 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T16:00:18.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T16:00:18.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:00:18.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:00:18.868 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:18.866+0000 7f16f082e640 -1 osd.0 0 log_to_monitors true 2024-08-19T16:00:19.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:00:20.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:00:20.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:00:20.087 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T16:00:20.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T16:00:20.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:00:20.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:00:20.361 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1064052537,v1:127.0.0.1:6803/1064052537] [v2:127.0.0.1:6804/1064052537,v1:127.0.0.1:6805/1064052537] exists,up 138b2521-7ff2-424c-abe4-d626ef1b2b9c 2024-08-19T16:00:20.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T16:00:20.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T16:00:20.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T16:00:20.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:547: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T16:00:20.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 1 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 2024-08-19T16:00:20.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-08-19T16:00:20.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T16:00:20.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T16:00:20.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T16:00:20.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/1 2024-08-19T16:00:20.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-08-19T16:00:20.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T16:00:20.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T16:00:20.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T16:00:20.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-08-19T16:00:20.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-08-19T16:00:20.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T16:00:20.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T16:00:20.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-08-19T16:00:20.373 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T16:00:20.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T16:00:20.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T16:00:20.374 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T16:00:20.374 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:00:20.374 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:00:20.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T16:00:20.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T16:00:20.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T16:00:20.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T16:00:20.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T16:00:20.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-08-19T16:00:20.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-08-19T16:00:20.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T16:00:20.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T16:00:20.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T16:00:20.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T16:00:20.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T16:00:20.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2' 2024-08-19T16:00:20.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/1 2024-08-19T16:00:20.377 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T16:00:20.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6239a07a-b88d-4601-bc4e-9be855da6431 2024-08-19T16:00:20.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6239a07a-b88d-4601-bc4e-9be855da6431' 2024-08-19T16:00:20.378 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 6239a07a-b88d-4601-bc4e-9be855da6431 2024-08-19T16:00:20.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T16:00:20.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAUbMNmDQ+4FhAA5ttjcL4rLoJ4geZBwvYh9A== 2024-08-19T16:00:20.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAUbMNmDQ+4FhAA5ttjcL4rLoJ4geZBwvYh9A=="}' 2024-08-19T16:00:20.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6239a07a-b88d-4601-bc4e-9be855da6431 -i td/osd-scrub-test/1/new.json 2024-08-19T16:00:20.688 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T16:00:20.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/1/new.json 2024-08-19T16:00:20.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 --mkfs --key AQAUbMNmDQ+4FhAA5ttjcL4rLoJ4geZBwvYh9A== --osd-uuid 6239a07a-b88d-4601-bc4e-9be855da6431 2024-08-19T16:00:20.731 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:20.729+0000 7fbcd9ddc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:00:20.764 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:20.763+0000 7fbcd9ddc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:00:20.765 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:20.764+0000 7fbcd9ddc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:00:20.765 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:20.764+0000 7fbcd9ddc640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-08-19T16:00:20.766 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:20.764+0000 7fbcd9ddc640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-08-19T16:00:20.766 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:20.764+0000 7fbcd9ddc640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-08-19T16:00:20.826 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:20.824+0000 7fbcd9ddc640 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-08-19T16:00:26.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-08-19T16:00:26.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T16:00:26.715 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T16:00:26.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T16:00:26.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T16:00:27.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T16:00:27.061 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T16:00:27.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 2024-08-19T16:00:27.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T16:00:27.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T16:00:27.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T16:00:27.091 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:27.089+0000 7fe1987b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:00:27.093 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:27.091+0000 7fe1987b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:00:27.094 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:27.092+0000 7fe1987b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:00:27.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T16:00:27.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T16:00:27.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T16:00:27.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T16:00:27.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T16:00:27.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:00:27.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T16:00:27.327 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T16:00:27.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:00:27.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:00:27.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:00:27.628 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:27.627+0000 7fe1987b6640 -1 Falling back to public interface 2024-08-19T16:00:28.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:00:28.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:00:28.594 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T16:00:28.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T16:00:28.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:00:28.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:00:28.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:00:29.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:00:29.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:00:29.868 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T16:00:29.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T16:00:29.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:00:29.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:00:30.080 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:30.078+0000 7fe1987b6640 -1 osd.1 0 log_to_monitors true 2024-08-19T16:00:30.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:00:31.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:00:31.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:00:31.206 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T16:00:31.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T16:00:31.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:00:31.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:00:31.475 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2179808961,v1:127.0.0.1:6811/2179808961] [v2:127.0.0.1:6812/2179808961,v1:127.0.0.1:6813/2179808961] exists,up 6239a07a-b88d-4601-bc4e-9be855da6431 2024-08-19T16:00:31.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T16:00:31.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T16:00:31.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T16:00:31.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:547: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T16:00:31.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 2 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 2024-08-19T16:00:31.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-08-19T16:00:31.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T16:00:31.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T16:00:31.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T16:00:31.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/2 2024-08-19T16:00:31.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-08-19T16:00:31.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T16:00:31.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T16:00:31.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T16:00:31.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-08-19T16:00:31.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-08-19T16:00:31.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T16:00:31.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T16:00:31.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-08-19T16:00:31.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T16:00:31.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T16:00:31.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T16:00:31.481 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T16:00:31.481 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:00:31.481 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:00:31.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T16:00:31.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T16:00:31.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T16:00:31.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T16:00:31.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T16:00:31.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-08-19T16:00:31.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-08-19T16:00:31.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T16:00:31.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T16:00:31.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T16:00:31.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T16:00:31.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T16:00:31.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2' 2024-08-19T16:00:31.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/2 2024-08-19T16:00:31.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T16:00:31.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1af7ae38-b763-4158-a97f-392aa0921386 2024-08-19T16:00:31.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1af7ae38-b763-4158-a97f-392aa0921386' 2024-08-19T16:00:31.483 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 1af7ae38-b763-4158-a97f-392aa0921386 2024-08-19T16:00:31.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T16:00:31.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAfbMNmJ1RsHRAAo7D4KXVoSAHqIk4WdFSMpg== 2024-08-19T16:00:31.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAfbMNmJ1RsHRAAo7D4KXVoSAHqIk4WdFSMpg=="}' 2024-08-19T16:00:31.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1af7ae38-b763-4158-a97f-392aa0921386 -i td/osd-scrub-test/2/new.json 2024-08-19T16:00:31.808 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T16:00:31.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/2/new.json 2024-08-19T16:00:31.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 --mkfs --key AQAfbMNmJ1RsHRAAo7D4KXVoSAHqIk4WdFSMpg== --osd-uuid 1af7ae38-b763-4158-a97f-392aa0921386 2024-08-19T16:00:31.852 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:31.850+0000 7fd849918640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:00:31.884 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:31.883+0000 7fd849918640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:00:31.885 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:31.884+0000 7fd849918640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:00:31.886 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:31.884+0000 7fd849918640 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-08-19T16:00:31.886 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:31.884+0000 7fd849918640 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-08-19T16:00:31.886 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:31.884+0000 7fd849918640 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-08-19T16:00:31.959 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:31.958+0000 7fd849918640 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-08-19T16:00:37.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-08-19T16:00:37.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T16:00:37.665 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T16:00:37.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T16:00:37.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T16:00:38.010 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T16:00:38.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T16:00:38.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 2024-08-19T16:00:38.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T16:00:38.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T16:00:38.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T16:00:38.040 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:38.038+0000 7f38df410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:00:38.042 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:38.040+0000 7f38df410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:00:38.043 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:38.041+0000 7f38df410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:00:38.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T16:00:38.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T16:00:38.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T16:00:38.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T16:00:38.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T16:00:38.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:00:38.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T16:00:38.273 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T16:00:38.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:00:38.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T16:00:38.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:00:38.559 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:38.558+0000 7f38df410640 -1 Falling back to public interface 2024-08-19T16:00:39.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:00:39.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:00:39.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T16:00:39.535 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T16:00:39.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:00:39.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T16:00:39.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:00:40.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:00:40.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:00:40.807 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T16:00:40.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T16:00:40.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:00:40.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T16:00:40.932 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:00:40.930+0000 7f38df410640 -1 osd.2 0 log_to_monitors true 2024-08-19T16:00:41.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:00:42.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:00:42.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:00:42.125 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T16:00:42.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T16:00:42.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:00:42.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T16:00:42.397 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3149655193,v1:127.0.0.1:6819/3149655193] [v2:127.0.0.1:6820/3149655193,v1:127.0.0.1:6821/3149655193] exists,up 1af7ae38-b763-4158-a97f-392aa0921386 2024-08-19T16:00:42.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T16:00:42.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T16:00:42.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T16:00:42.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:553: TEST_dump_scrub_schedule: create_pool test 1 1 2024-08-19T16:00:42.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-08-19T16:00:42.790 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' created 2024-08-19T16:00:42.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T16:00:43.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:554: TEST_dump_scrub_schedule: wait_for_clean 2024-08-19T16:00:43.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T16:00:43.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T16:00:43.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T16:00:43.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T16:00:43.804 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T16:00:43.804 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T16:00:43.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T16:00:43.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T16:00:43.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T16:00:43.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T16:00:43.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T16:00:43.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T16:00:43.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T16:00:43.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=500 2024-08-19T16:00:43.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T16:00:44.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T16:00:44.176 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-08-19T16:00:44.176 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-08-19T16:00:44.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T16:00:44.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T16:00:44.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T16:00:44.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836487 2024-08-19T16:00:44.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836487 2024-08-19T16:00:44.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487' 2024-08-19T16:00:44.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T16:00:44.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T16:00:44.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-08-19T16:00:44.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-08-19T16:00:44.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2024-08-19T16:00:44.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T16:00:44.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T16:00:44.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509442 2024-08-19T16:00:44.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509442 2024-08-19T16:00:44.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672964 2-64424509442' 2024-08-19T16:00:44.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T16:00:44.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836487 2024-08-19T16:00:44.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T16:00:44.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T16:00:44.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836487 2024-08-19T16:00:44.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T16:00:44.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836487 2024-08-19T16:00:44.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-08-19T16:00:44.604 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836487 2024-08-19T16:00:44.604 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T16:00:44.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836486 -lt 21474836487 2024-08-19T16:00:44.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T16:00:45.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 500 -eq 0 ']' 2024-08-19T16:00:45.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T16:00:46.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836487 2024-08-19T16:00:46.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T16:00:46.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-08-19T16:00:46.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T16:00:46.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T16:00:46.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-08-19T16:00:46.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T16:00:46.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-08-19T16:00:46.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-08-19T16:00:46.156 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672964 2024-08-19T16:00:46.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T16:00:46.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672964 2024-08-19T16:00:46.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T16:00:46.420 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509442 2024-08-19T16:00:46.420 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T16:00:46.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-19T16:00:46.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509442 2024-08-19T16:00:46.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T16:00:46.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509442 2024-08-19T16:00:46.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-08-19T16:00:46.422 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509442 2024-08-19T16:00:46.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T16:00:46.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509442 -lt 64424509442 2024-08-19T16:00:46.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T16:00:46.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T16:00:46.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T16:00:47.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T16:00:47.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T16:00:47.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T16:00:47.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T16:00:47.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T16:00:47.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T16:00:47.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T16:00:47.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T16:00:47.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T16:00:47.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T16:00:47.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T16:00:47.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T16:00:47.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T16:00:47.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T16:00:47.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T16:00:47.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:555: TEST_dump_scrub_schedule: ceph osd dump 2024-08-19T16:00:47.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:555: TEST_dump_scrub_schedule: grep '^pool.*['\'']test['\'']' 2024-08-19T16:00:47.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:555: TEST_dump_scrub_schedule: awk '{ print $2 }' 2024-08-19T16:00:47.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:555: TEST_dump_scrub_schedule: poolid=1 2024-08-19T16:00:47.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:557: TEST_dump_scrub_schedule: dd if=/dev/urandom of=testdata.536210 bs=1032 count=1 2024-08-19T16:00:47.879 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records in 2024-08-19T16:00:47.879 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records out 2024-08-19T16:00:47.879 INFO:tasks.workunit.client.0.smithi102.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.5946e-05 s, 22.5 MB/s 2024-08-19T16:00:47.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: seq 1 15 2024-08-19T16:00:47.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-08-19T16:00:47.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj1 testdata.536210 2024-08-19T16:00:48.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-08-19T16:00:48.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj2 testdata.536210 2024-08-19T16:00:48.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-08-19T16:00:48.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj3 testdata.536210 2024-08-19T16:00:48.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-08-19T16:00:48.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj4 testdata.536210 2024-08-19T16:00:48.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-08-19T16:00:48.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj5 testdata.536210 2024-08-19T16:00:48.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-08-19T16:00:48.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj6 testdata.536210 2024-08-19T16:00:48.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-08-19T16:00:48.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj7 testdata.536210 2024-08-19T16:00:48.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-08-19T16:00:48.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj8 testdata.536210 2024-08-19T16:00:49.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-08-19T16:00:49.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj9 testdata.536210 2024-08-19T16:00:49.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-08-19T16:00:49.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj10 testdata.536210 2024-08-19T16:00:49.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-08-19T16:00:49.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj11 testdata.536210 2024-08-19T16:00:49.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-08-19T16:00:49.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj12 testdata.536210 2024-08-19T16:00:49.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-08-19T16:00:49.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj13 testdata.536210 2024-08-19T16:00:49.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-08-19T16:00:49.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj14 testdata.536210 2024-08-19T16:00:49.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-08-19T16:00:49.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj15 testdata.536210 2024-08-19T16:00:50.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rm -f testdata.536210 2024-08-19T16:00:50.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:564: TEST_dump_scrub_schedule: local pgid=1.0 2024-08-19T16:00:50.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: date -Ins 2024-08-19T16:00:50.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: local now_is=2024-08-19T16:00:50,084683628+00:00 2024-08-19T16:00:50.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:573: TEST_dump_scrub_schedule: expct_starting=(['query_active']='false' ['query_is_future']='true' ['query_schedule']='scrub scheduled') 2024-08-19T16:00:50.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:573: TEST_dump_scrub_schedule: declare -A expct_starting 2024-08-19T16:00:50.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:574: TEST_dump_scrub_schedule: declare -A sched_data 2024-08-19T16:00:50.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:575: TEST_dump_scrub_schedule: extract_published_sch 1.0 2024-08-19T16:00:50,084683628+00:00 2019-10-12T20:32:43.645168+0000 sched_data 2024-08-19T16:00:50.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.0 2024-08-19T16:00:50.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sched_data 2024-08-19T16:00:50.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2024-08-19T16:00:50,084683628+00:00 2024-08-19T16:00:50.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2019-10-12T20:32:43.645168+0000 2024-08-19T16:00:50.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=1 2024-08-19T16:00:50.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2024-08-19T16:00:50.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2024-08-19T16:00:50.334 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:00:50.458 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:00:50.458 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=19 2024-08-19T16:00:50.458 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=29 2024-08-19T16:00:50.458 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:00:50.458 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:00:50.458 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-20T16:00:42.739 (2024-08-20T16:00:42.739)' 2024-08-19T16:00:50.458 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=0 2024-08-19T16:00:50.458 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-19T16:00:42.739035+0000' 2024-08-19T16:00:50.458 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='0x0' 2024-08-19T16:00:50.458 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:00:50.459 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:00:50.459 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=null 2024-08-19T16:00:50.459 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:00:50.459 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:00:50.459 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=0 2024-08-19T16:00:50.459 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:00:50.459 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-20T16:00:42.739035+0000' 2024-08-19T16:00:50.459 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:00:50.459 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:00:50.459 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=19 2024-08-19T16:00:50.459 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=13 2024-08-19T16:00:50.459 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:00:50.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:576: TEST_dump_scrub_schedule: schedule_against_expected sched_data expct_starting initial 2024-08-19T16:00:50.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:170: schedule_against_expected: local -n dict=sched_data 2024-08-19T16:00:50.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:171: schedule_against_expected: local -n ep=expct_starting 2024-08-19T16:00:50.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:172: schedule_against_expected: local extr_dbg=1 2024-08-19T16:00:50.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:175: schedule_against_expected: local saved_echo_flag=x 2024-08-19T16:00:50.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:176: schedule_against_expected: set +x 2024-08-19T16:00:50.462 INFO:tasks.workunit.client.0.smithi102.stdout:-- - comparing: 2024-08-19T16:00:50.463 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_schedule expected: scrub scheduled in actual: scrub scheduled 2024-08-19T16:00:50.463 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_is_future expected: true in actual: true 2024-08-19T16:00:50.463 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_active expected: false in actual: false 2024-08-19T16:00:50.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:195: schedule_against_expected: return 0 2024-08-19T16:00:50.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:577: TEST_dump_scrub_schedule: (( 0 == 0 )) 2024-08-19T16:00:50.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:578: TEST_dump_scrub_schedule: echo 'last-scrub --- ' 0x0 2024-08-19T16:00:50.464 INFO:tasks.workunit.client.0.smithi102.stdout:last-scrub --- 0x0 2024-08-19T16:00:50.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:584: TEST_dump_scrub_schedule: saved_last_stamp=2024-08-19T16:00:42.739035+0000 2024-08-19T16:00:50.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:585: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 0 2024-08-19T16:00:50.555 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: { 2024-08-19T16:00:50.555 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_scrub_sleep = '' " 2024-08-19T16:00:50.555 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:00:50.568 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: { 2024-08-19T16:00:50.568 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_scrub_sleep = '' " 2024-08-19T16:00:50.568 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:00:50.581 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: { 2024-08-19T16:00:50.581 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_scrub_sleep = '' " 2024-08-19T16:00:50.581 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:00:50.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:586: TEST_dump_scrub_schedule: ceph tell 1.0 deep-scrub 2024-08-19T16:00:50.699 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T16:00:50.699 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": true, 2024-08-19T16:00:50.700 INFO:tasks.workunit.client.0.smithi102.stdout: "must": true, 2024-08-19T16:00:50.700 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "0.000000" 2024-08-19T16:00:50.700 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:00:50.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:591: TEST_dump_scrub_schedule: sleep 5 2024-08-19T16:00:55.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:592: TEST_dump_scrub_schedule: sched_data=() 2024-08-19T16:00:55.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:593: TEST_dump_scrub_schedule: expct_qry_duration=(['query_last_duration']='0' ['query_last_duration_neg']='not0') 2024-08-19T16:00:55.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:593: TEST_dump_scrub_schedule: declare -A expct_qry_duration 2024-08-19T16:00:55.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:594: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-08-19T16:00:42.739035+0000 expct_qry_duration 'WaitingAfterScrub ' sched_data 2024-08-19T16:00:55.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-08-19T16:00:55.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-08-19T16:00:55.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-08-19T16:00:42.739035+0000 2024-08-19T16:00:55.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=expct_qry_duration 2024-08-19T16:00:55.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-08-19T16:00:55.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-08-19T16:00:55.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-08-19T16:00:55.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-08-19T16:00:55.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-08-19T16:00:55.712 INFO:tasks.workunit.client.0.smithi102.stdout:waiting for any condition (WaitingAfterScrub ): pg:1.0 dt:2024-08-19T16:00:42.739035+0000 (10 retries) 2024-08-19T16:00:56.472 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:00:56.591 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:00:56.591 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=19 2024-08-19T16:00:56.591 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=40 2024-08-19T16:00:56.591 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:00:56.591 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:00:56.591 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-20T16:00:51.117 (2024-08-20T16:00:51.117)' 2024-08-19T16:00:56.591 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=1 2024-08-19T16:00:56.591 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-19T16:00:51.117519+0000' 2024-08-19T16:00:56.591 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='19x15' 2024-08-19T16:00:56.591 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:00:56.591 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:00:56.591 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=null 2024-08-19T16:00:56.592 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:00:56.592 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:00:56.592 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=1 2024-08-19T16:00:56.592 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:00:56.592 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-20T16:00:51.117519+0000' 2024-08-19T16:00:56.592 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:00:56.592 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:00:56.592 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=19 2024-08-19T16:00:56.592 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=38 2024-08-19T16:00:56.592 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:00:56.594 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 1 ~ false / 40 / 38 / true / 2024-08-19T16:00:51.117519+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-08-19T16:00:56.594 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 1 2024-08-19T16:00:56.595 INFO:tasks.workunit.client.0.smithi102.stdout:WaitingAfterScrub - 'query_last_duration' actual value (1) matches expected (0) (negation: 1) 2024-08-19T16:00:56.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-08-19T16:00:56.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:596: TEST_dump_scrub_schedule: sched_data=() 2024-08-19T16:00:56.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:597: TEST_dump_scrub_schedule: expct_dmp_duration=(['dmp_last_duration']='0' ['dmp_last_duration_neg']='not0') 2024-08-19T16:00:56.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:597: TEST_dump_scrub_schedule: declare -A expct_dmp_duration 2024-08-19T16:00:56.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:598: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-08-19T16:00:42.739035+0000 expct_dmp_duration 'WaitingAfterScrub_dmp ' sched_data 2024-08-19T16:00:56.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-08-19T16:00:56.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-08-19T16:00:56.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-08-19T16:00:42.739035+0000 2024-08-19T16:00:56.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=expct_dmp_duration 2024-08-19T16:00:56.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-08-19T16:00:56.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-08-19T16:00:56.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-08-19T16:00:56.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-08-19T16:00:56.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-08-19T16:00:56.597 INFO:tasks.workunit.client.0.smithi102.stdout:waiting for any condition (WaitingAfterScrub_dmp ): pg:1.0 dt:2024-08-19T16:00:42.739035+0000 (10 retries) 2024-08-19T16:00:57.346 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:00:57.463 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:00:57.463 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=19 2024-08-19T16:00:57.464 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=40 2024-08-19T16:00:57.464 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:00:57.464 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:00:57.464 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-20T16:00:51.117 (2024-08-20T16:00:51.117)' 2024-08-19T16:00:57.464 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=1 2024-08-19T16:00:57.464 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-19T16:00:51.117519+0000' 2024-08-19T16:00:57.464 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='19x15' 2024-08-19T16:00:57.464 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:00:57.464 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:00:57.464 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=null 2024-08-19T16:00:57.464 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:00:57.464 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:00:57.464 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=1 2024-08-19T16:00:57.465 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:00:57.465 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-20T16:00:51.117519+0000' 2024-08-19T16:00:57.465 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:00:57.465 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:00:57.465 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=19 2024-08-19T16:00:57.465 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=38 2024-08-19T16:00:57.465 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:00:57.467 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 1 ~ false / 40 / 38 / true / 2024-08-19T16:00:51.117519+0000 / scrub scheduled %%% dmp_last_duration dmp_last_duration_neg 2024-08-19T16:00:57.467 INFO:tasks.workunit.client.0.smithi102.stdout:key is dmp_last_duration: negation:1 # expected: 0 # in actual: 1 2024-08-19T16:00:57.467 INFO:tasks.workunit.client.0.smithi102.stdout:WaitingAfterScrub_dmp - 'dmp_last_duration' actual value (1) matches expected (0) (negation: 1) 2024-08-19T16:00:57.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-08-19T16:00:57.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:600: TEST_dump_scrub_schedule: sleep 2 2024-08-19T16:00:59.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:607: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_chunk_max 3 2024-08-19T16:00:59.569 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: { 2024-08-19T16:00:59.570 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_scrub_chunk_max = '' (not observed, change may require restart) " 2024-08-19T16:00:59.570 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:00:59.583 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: { 2024-08-19T16:00:59.583 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_scrub_chunk_max = '' (not observed, change may require restart) " 2024-08-19T16:00:59.583 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:00:59.596 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: { 2024-08-19T16:00:59.596 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_scrub_chunk_max = '' (not observed, change may require restart) " 2024-08-19T16:00:59.596 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:00:59.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:608: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 1.0 2024-08-19T16:00:59.746 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: { 2024-08-19T16:00:59.746 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_scrub_sleep = '' " 2024-08-19T16:00:59.746 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:00:59.759 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: { 2024-08-19T16:00:59.759 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_scrub_sleep = '' " 2024-08-19T16:00:59.759 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:00:59.772 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: { 2024-08-19T16:00:59.772 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_scrub_sleep = '' " 2024-08-19T16:00:59.772 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:00:59.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:609: TEST_dump_scrub_schedule: ceph osd set noscrub 2024-08-19T16:01:00.207 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T16:01:00.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:610: TEST_dump_scrub_schedule: sleep 2 2024-08-19T16:01:02.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:611: TEST_dump_scrub_schedule: saved_last_stamp=2024-08-19T16:00:51.117519+0000 2024-08-19T16:01:02.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:613: TEST_dump_scrub_schedule: ceph tell 1.0 schedule-scrub 2024-08-19T16:01:02.321 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T16:01:02.321 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": false, 2024-08-19T16:01:02.321 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T16:01:02.321 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-12T15:59:22.320407+0000" 2024-08-19T16:01:02.321 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:01:02.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:614: TEST_dump_scrub_schedule: sleep 1 2024-08-19T16:01:03.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:615: TEST_dump_scrub_schedule: sched_data=() 2024-08-19T16:01:03.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:616: TEST_dump_scrub_schedule: expct_scrub_peri_sched=(['query_is_future']='false') 2024-08-19T16:01:03.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:616: TEST_dump_scrub_schedule: declare -A expct_scrub_peri_sched 2024-08-19T16:01:03.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:617: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-08-19T16:00:51.117519+0000 expct_scrub_peri_sched waitingBeingScheduled sched_data 2024-08-19T16:01:03.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-08-19T16:01:03.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-08-19T16:01:03.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-08-19T16:00:51.117519+0000 2024-08-19T16:01:03.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=expct_scrub_peri_sched 2024-08-19T16:01:03.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-08-19T16:01:03.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-08-19T16:01:03.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-08-19T16:01:03.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-08-19T16:01:03.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-08-19T16:01:03.334 INFO:tasks.workunit.client.0.smithi102.stdout:waiting for any condition (waitingBeingScheduled): pg:1.0 dt:2024-08-19T16:00:51.117519+0000 (10 retries) 2024-08-19T16:01:04.112 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:01:04.229 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:01:04.229 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=20 2024-08-19T16:01:04.229 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=45 2024-08-19T16:01:04.229 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:01:04.229 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:01:04.230 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-19T16:01:08.155 (2024-08-13T15:59:22.320)' 2024-08-19T16:01:04.230 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=1 2024-08-19T16:01:04.230 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-12T15:59:22.320407+0000' 2024-08-19T16:01:04.230 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='19x15' 2024-08-19T16:01:04.230 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:01:04.230 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:01:04.230 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=null 2024-08-19T16:01:04.230 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:01:04.230 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:01:04.230 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=1 2024-08-19T16:01:04.230 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:01:04.230 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-20T16:00:51.117519+0000' 2024-08-19T16:01:04.230 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:01:04.231 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:01:04.231 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=20 2024-08-19T16:01:04.231 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=42 2024-08-19T16:01:04.231 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:01:04.233 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 1 ~ false / 45 / 42 / true / 2024-08-12T15:59:22.320407+0000 / scrub scheduled %%% query_is_future 2024-08-19T16:01:04.233 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_is_future: negation:0 # expected: false # in actual: true 2024-08-19T16:01:04.994 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:01:05.113 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:01:05.113 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=20 2024-08-19T16:01:05.114 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=45 2024-08-19T16:01:05.114 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:01:05.114 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:01:05.114 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-19T16:01:08.155 (2024-08-13T15:59:22.320)' 2024-08-19T16:01:05.114 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=1 2024-08-19T16:01:05.114 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-12T15:59:22.320407+0000' 2024-08-19T16:01:05.114 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='19x15' 2024-08-19T16:01:05.114 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:01:05.114 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:01:05.114 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=null 2024-08-19T16:01:05.115 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:01:05.115 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:01:05.115 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=1 2024-08-19T16:01:05.115 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:01:05.115 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-20T16:00:51.117519+0000' 2024-08-19T16:01:05.115 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:01:05.115 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:01:05.115 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=20 2024-08-19T16:01:05.115 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=42 2024-08-19T16:01:05.115 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:01:05.117 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 2 ~ false / 45 / 42 / true / 2024-08-12T15:59:22.320407+0000 / scrub scheduled %%% query_is_future 2024-08-19T16:01:05.117 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_is_future: negation:0 # expected: false # in actual: true 2024-08-19T16:01:05.894 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:01:06.011 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:01:06.011 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=20 2024-08-19T16:01:06.011 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=45 2024-08-19T16:01:06.011 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:01:06.011 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:01:06.011 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-19T16:01:08.155 (2024-08-13T15:59:22.320)' 2024-08-19T16:01:06.011 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=1 2024-08-19T16:01:06.011 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-12T15:59:22.320407+0000' 2024-08-19T16:01:06.011 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='19x15' 2024-08-19T16:01:06.012 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:01:06.012 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:01:06.012 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=null 2024-08-19T16:01:06.012 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:01:06.012 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:01:06.012 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=1 2024-08-19T16:01:06.012 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:01:06.012 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-20T16:00:51.117519+0000' 2024-08-19T16:01:06.012 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:01:06.012 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:01:06.012 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=20 2024-08-19T16:01:06.012 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=42 2024-08-19T16:01:06.012 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:01:06.014 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 3 ~ false / 45 / 42 / true / 2024-08-12T15:59:22.320407+0000 / scrub scheduled %%% query_is_future 2024-08-19T16:01:06.014 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_is_future: negation:0 # expected: false # in actual: true 2024-08-19T16:01:06.765 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:01:06.886 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:01:06.886 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=20 2024-08-19T16:01:06.886 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=46 2024-08-19T16:01:06.886 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:01:06.886 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:01:06.886 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-19T16:01:08.155 (2024-08-13T15:59:22.320)' 2024-08-19T16:01:06.886 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=1 2024-08-19T16:01:06.886 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-12T15:59:22.320407+0000' 2024-08-19T16:01:06.886 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='19x15' 2024-08-19T16:01:06.887 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:01:06.887 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:01:06.887 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=null 2024-08-19T16:01:06.887 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:01:06.887 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:01:06.887 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=1 2024-08-19T16:01:06.887 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:01:06.887 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-20T16:00:51.117519+0000' 2024-08-19T16:01:06.887 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:01:06.887 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:01:06.887 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=20 2024-08-19T16:01:06.887 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=42 2024-08-19T16:01:06.887 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:01:06.889 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 4 ~ false / 46 / 42 / true / 2024-08-12T15:59:22.320407+0000 / scrub scheduled %%% query_is_future 2024-08-19T16:01:06.889 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_is_future: negation:0 # expected: false # in actual: true 2024-08-19T16:01:07.642 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:01:07.761 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:01:07.761 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=20 2024-08-19T16:01:07.761 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=46 2024-08-19T16:01:07.761 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:01:07.761 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:01:07.761 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-19T16:01:08.155 (2024-08-13T15:59:22.320)' 2024-08-19T16:01:07.761 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=1 2024-08-19T16:01:07.762 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-12T15:59:22.320407+0000' 2024-08-19T16:01:07.762 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='19x15' 2024-08-19T16:01:07.762 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:01:07.762 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:01:07.762 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=null 2024-08-19T16:01:07.762 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:01:07.762 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:01:07.762 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=1 2024-08-19T16:01:07.762 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:01:07.762 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-20T16:00:51.117519+0000' 2024-08-19T16:01:07.763 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:01:07.763 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:01:07.763 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=20 2024-08-19T16:01:07.763 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=42 2024-08-19T16:01:07.763 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:01:07.764 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 5 ~ false / 46 / 42 / true / 2024-08-12T15:59:22.320407+0000 / scrub scheduled %%% query_is_future 2024-08-19T16:01:07.765 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_is_future: negation:0 # expected: false # in actual: true 2024-08-19T16:01:08.515 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:01:08.633 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:01:08.633 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=20 2024-08-19T16:01:08.633 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=46 2024-08-19T16:01:08.634 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:01:08.634 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='queued for scrub' 2024-08-19T16:01:08.634 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='0' 2024-08-19T16:01:08.634 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=1 2024-08-19T16:01:08.634 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-12T15:59:22.320407+0000' 2024-08-19T16:01:08.634 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='19x15' 2024-08-19T16:01:08.634 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=false 2024-08-19T16:01:08.634 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=false 2024-08-19T16:01:08.634 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=null 2024-08-19T16:01:08.634 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:01:08.634 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:01:08.635 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=1 2024-08-19T16:01:08.635 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:01:08.635 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-19T16:01:08.155436+0000' 2024-08-19T16:01:08.635 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:01:08.635 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:01:08.635 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=20 2024-08-19T16:01:08.635 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=45 2024-08-19T16:01:08.635 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:01:08.637 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 6 ~ false / 46 / 45 / false / 2024-08-12T15:59:22.320407+0000 / queued for scrub %%% query_is_future 2024-08-19T16:01:08.637 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_is_future: negation:0 # expected: false # in actual: false 2024-08-19T16:01:08.637 INFO:tasks.workunit.client.0.smithi102.stdout:waitingBeingScheduled - 'query_is_future' actual value (false) matches expected (false) (negation: 0) 2024-08-19T16:01:08.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-08-19T16:01:08.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:629: TEST_dump_scrub_schedule: saved_last_stamp=2024-08-12T15:59:22.320407+0000 2024-08-19T16:01:08.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:630: TEST_dump_scrub_schedule: ceph osd unset noscrub 2024-08-19T16:01:09.115 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is unset 2024-08-19T16:01:09.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:632: TEST_dump_scrub_schedule: cond_active=(['query_active']='true') 2024-08-19T16:01:09.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:632: TEST_dump_scrub_schedule: declare -A cond_active 2024-08-19T16:01:09.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:633: TEST_dump_scrub_schedule: sched_data=() 2024-08-19T16:01:09.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:634: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-08-12T15:59:22.320407+0000 cond_active 'WaitingActive ' sched_data 2024-08-19T16:01:09.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-08-19T16:01:09.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-08-19T16:01:09.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-08-12T15:59:22.320407+0000 2024-08-19T16:01:09.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=cond_active 2024-08-19T16:01:09.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-08-19T16:01:09.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-08-19T16:01:09.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-08-19T16:01:09.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-08-19T16:01:09.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-08-19T16:01:09.129 INFO:tasks.workunit.client.0.smithi102.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2024-08-12T15:59:22.320407+0000 (10 retries) 2024-08-19T16:01:09.891 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:01:10.010 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:01:10.010 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=20 2024-08-19T16:01:10.010 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=46 2024-08-19T16:01:10.010 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:01:10.010 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:01:10.010 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-19T16:01:14.172 (2024-08-13T15:59:22.320)' 2024-08-19T16:01:10.010 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=1 2024-08-19T16:01:10.010 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-12T15:59:22.320407+0000' 2024-08-19T16:01:10.010 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='19x15' 2024-08-19T16:01:10.011 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:01:10.011 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:01:10.011 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=null 2024-08-19T16:01:10.011 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:01:10.011 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:01:10.011 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=1 2024-08-19T16:01:10.011 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:01:10.011 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-19T16:01:08.155436+0000' 2024-08-19T16:01:10.011 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=false 2024-08-19T16:01:10.011 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:01:10.011 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=20 2024-08-19T16:01:10.011 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=45 2024-08-19T16:01:10.011 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:01:10.013 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 1 ~ false / 46 / 45 / true / 2024-08-12T15:59:22.320407+0000 / scrub scheduled %%% query_active 2024-08-19T16:01:10.013 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_active: negation:0 # expected: true # in actual: false 2024-08-19T16:01:10.770 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:01:10.909 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:01:10.909 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=21 2024-08-19T16:01:10.909 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=48 2024-08-19T16:01:10.909 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:01:10.909 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:01:10.909 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-19T16:01:14.172 (2024-08-13T15:59:22.320)' 2024-08-19T16:01:10.910 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=1 2024-08-19T16:01:10.910 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-12T15:59:22.320407+0000' 2024-08-19T16:01:10.910 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='19x15' 2024-08-19T16:01:10.910 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:01:10.910 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:01:10.910 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=null 2024-08-19T16:01:10.910 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:01:10.910 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:01:10.910 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=1 2024-08-19T16:01:10.910 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:01:10.910 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-19T16:01:08.155436+0000' 2024-08-19T16:01:10.910 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=false 2024-08-19T16:01:10.910 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:01:10.911 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=20 2024-08-19T16:01:10.911 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=45 2024-08-19T16:01:10.911 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:01:10.912 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 2 ~ false / 48 / 45 / true / 2024-08-12T15:59:22.320407+0000 / scrub scheduled %%% query_active 2024-08-19T16:01:10.912 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_active: negation:0 # expected: true # in actual: false 2024-08-19T16:01:11.670 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:01:11.788 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:01:11.788 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=21 2024-08-19T16:01:11.788 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=49 2024-08-19T16:01:11.788 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:01:11.788 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:01:11.788 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-19T16:01:14.172 (2024-08-13T15:59:22.320)' 2024-08-19T16:01:11.788 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=1 2024-08-19T16:01:11.788 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-12T15:59:22.320407+0000' 2024-08-19T16:01:11.788 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='19x15' 2024-08-19T16:01:11.788 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:01:11.789 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:01:11.789 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=null 2024-08-19T16:01:11.789 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:01:11.789 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:01:11.789 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=1 2024-08-19T16:01:11.789 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:01:11.789 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-19T16:01:14.172103+0000' 2024-08-19T16:01:11.789 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:01:11.789 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:01:11.789 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=21 2024-08-19T16:01:11.789 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=48 2024-08-19T16:01:11.789 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:01:11.791 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 3 ~ false / 49 / 48 / true / 2024-08-12T15:59:22.320407+0000 / scrub scheduled %%% query_active 2024-08-19T16:01:11.791 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_active: negation:0 # expected: true # in actual: false 2024-08-19T16:01:12.546 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:01:12.663 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:01:12.663 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=21 2024-08-19T16:01:12.663 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=49 2024-08-19T16:01:12.664 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:01:12.664 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:01:12.664 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-19T16:01:14.172 (2024-08-13T15:59:22.320)' 2024-08-19T16:01:12.664 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=1 2024-08-19T16:01:12.664 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-12T15:59:22.320407+0000' 2024-08-19T16:01:12.664 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='19x15' 2024-08-19T16:01:12.664 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:01:12.664 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:01:12.664 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=null 2024-08-19T16:01:12.664 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:01:12.664 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:01:12.664 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=1 2024-08-19T16:01:12.665 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:01:12.665 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-19T16:01:14.172103+0000' 2024-08-19T16:01:12.665 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:01:12.665 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:01:12.665 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=21 2024-08-19T16:01:12.665 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=48 2024-08-19T16:01:12.665 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:01:12.667 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 4 ~ false / 49 / 48 / true / 2024-08-12T15:59:22.320407+0000 / scrub scheduled %%% query_active 2024-08-19T16:01:12.667 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_active: negation:0 # expected: true # in actual: false 2024-08-19T16:01:13.417 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:01:13.534 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:01:13.534 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=21 2024-08-19T16:01:13.534 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=49 2024-08-19T16:01:13.534 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:01:13.534 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:01:13.535 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-19T16:01:14.172 (2024-08-13T15:59:22.320)' 2024-08-19T16:01:13.535 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=1 2024-08-19T16:01:13.535 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-12T15:59:22.320407+0000' 2024-08-19T16:01:13.535 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='19x15' 2024-08-19T16:01:13.535 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:01:13.535 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:01:13.535 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=null 2024-08-19T16:01:13.535 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:01:13.535 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:01:13.535 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=1 2024-08-19T16:01:13.535 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:01:13.535 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-19T16:01:14.172103+0000' 2024-08-19T16:01:13.535 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:01:13.535 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:01:13.536 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=21 2024-08-19T16:01:13.536 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=48 2024-08-19T16:01:13.536 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:01:13.537 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 5 ~ false / 49 / 48 / true / 2024-08-12T15:59:22.320407+0000 / scrub scheduled %%% query_active 2024-08-19T16:01:13.537 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_active: negation:0 # expected: true # in actual: false 2024-08-19T16:01:14.289 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:01:14.407 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:01:14.408 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=21 2024-08-19T16:01:14.408 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=52 2024-08-19T16:01:14.408 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=true 2024-08-19T16:01:14.408 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrubbing' 2024-08-19T16:01:14.408 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='0' 2024-08-19T16:01:14.408 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=1 2024-08-19T16:01:14.408 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-12T15:59:22.320407+0000' 2024-08-19T16:01:14.408 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='19x15' 2024-08-19T16:01:14.408 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=false 2024-08-19T16:01:14.408 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=false 2024-08-19T16:01:14.409 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=null 2024-08-19T16:01:14.409 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:01:14.409 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:01:14.409 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=1 2024-08-19T16:01:14.409 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:01:14.409 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-19T16:01:14.172103+0000' 2024-08-19T16:01:14.409 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:01:14.409 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:01:14.409 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=21 2024-08-19T16:01:14.409 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=48 2024-08-19T16:01:14.410 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:01:14.411 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 6 ~ true / 52 / 48 / false / 2024-08-12T15:59:22.320407+0000 / scrubbing %%% query_active 2024-08-19T16:01:14.411 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_active: negation:0 # expected: true # in actual: true 2024-08-19T16:01:14.411 INFO:tasks.workunit.client.0.smithi102.stdout:WaitingActive - 'query_active' actual value (true) matches expected (true) (negation: 0) 2024-08-19T16:01:14.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-08-19T16:01:14.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:638: TEST_dump_scrub_schedule: cond_active_dmp=(['dmp_state_has_scrubbing']='true' ['query_active']='false') 2024-08-19T16:01:14.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:638: TEST_dump_scrub_schedule: declare -A cond_active_dmp 2024-08-19T16:01:14.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:639: TEST_dump_scrub_schedule: sched_data=() 2024-08-19T16:01:14.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:640: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-08-12T15:59:22.320407+0000 cond_active_dmp 'WaitingActive ' sched_data 2024-08-19T16:01:14.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-08-19T16:01:14.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-08-19T16:01:14.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-08-12T15:59:22.320407+0000 2024-08-19T16:01:14.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=cond_active_dmp 2024-08-19T16:01:14.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-08-19T16:01:14.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-08-19T16:01:14.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-08-19T16:01:14.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-08-19T16:01:14.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-08-19T16:01:14.415 INFO:tasks.workunit.client.0.smithi102.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2024-08-12T15:59:22.320407+0000 (10 retries) 2024-08-19T16:01:15.170 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:01:15.340 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:01:15.340 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=21 2024-08-19T16:01:15.340 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=57 2024-08-19T16:01:15.341 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:01:15.341 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:01:15.341 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-20T16:01:15.175 (2024-08-20T16:01:15.175)' 2024-08-19T16:01:15.341 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=2 2024-08-19T16:01:15.341 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-19T16:01:15.175855+0000' 2024-08-19T16:01:15.341 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='19x15' 2024-08-19T16:01:15.341 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:01:15.341 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:01:15.341 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=null 2024-08-19T16:01:15.341 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:01:15.341 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:01:15.342 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=1 2024-08-19T16:01:15.342 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:01:15.342 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-19T16:01:14.172103+0000' 2024-08-19T16:01:15.342 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:01:15.342 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:01:15.342 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=21 2024-08-19T16:01:15.342 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=48 2024-08-19T16:01:15.342 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:01:15.344 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 1 ~ false / 57 / 48 / true / 2024-08-19T16:01:15.175855+0000 / scrub scheduled %%% dmp_state_has_scrubbing query_active 2024-08-19T16:01:15.344 INFO:tasks.workunit.client.0.smithi102.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-08-19T16:01:15.344 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_active: negation:0 # expected: false # in actual: false 2024-08-19T16:01:15.344 INFO:tasks.workunit.client.0.smithi102.stdout:WaitingActive - 'query_active' actual value (false) matches expected (false) (negation: 0) 2024-08-19T16:01:15.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-08-19T16:01:15.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:641: TEST_dump_scrub_schedule: perf_counters td/osd-scrub-test 3 2024-08-19T16:01:15.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2024-08-19T16:01:15.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2024-08-19T16:01:15.346 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2024-08-19T16:01:15.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2024-08-19T16:01:15.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T16:01:15.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2024-08-19T16:01:15.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-08-19T16:01:15.439 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T16:01:15.440 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_dp_ec": [ 2024-08-19T16:01:15.440 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:01:15.440 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:01:15.440 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "deep", 2024-08-19T16:01:15.440 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "ec" 2024-08-19T16:01:15.440 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.440 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:01:15.440 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:01:15.440 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:01:15.440 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:01:15.440 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:01:15.440 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.441 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.441 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.441 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.441 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:01:15.441 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:01:15.441 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.441 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.441 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.441 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.441 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:01:15.441 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:01:15.441 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:01:15.441 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:01:15.442 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:01:15.442 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:01:15.442 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:01:15.442 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.442 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.442 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.442 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.442 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:01:15.442 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:01:15.442 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:01:15.442 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:01:15.442 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.443 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.443 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.443 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.443 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:01:15.443 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.443 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.443 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:01:15.443 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_dp_repl": [ 2024-08-19T16:01:15.443 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:01:15.443 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:01:15.443 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "deep", 2024-08-19T16:01:15.443 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "replicated" 2024-08-19T16:01:15.444 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.444 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:01:15.444 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:01:15.444 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:01:15.444 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:01:15.444 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:01:15.444 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.444 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.444 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.444 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.444 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:01:15.445 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:01:15.445 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.445 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.445 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.445 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.445 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:01:15.445 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:01:15.445 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:01:15.445 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:01:15.445 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:01:15.445 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:01:15.446 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:01:15.446 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.446 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.446 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.446 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.446 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:01:15.446 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:01:15.446 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:01:15.446 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:01:15.446 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.446 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.446 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.447 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.447 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:01:15.447 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.447 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.447 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:01:15.447 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_sh_ec": [ 2024-08-19T16:01:15.447 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:01:15.447 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:01:15.447 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "shallow", 2024-08-19T16:01:15.447 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "ec" 2024-08-19T16:01:15.447 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.447 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:01:15.448 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:01:15.448 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:01:15.448 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:01:15.448 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:01:15.448 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.448 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.448 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.448 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.448 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:01:15.448 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:01:15.448 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.448 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.448 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.449 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.449 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:01:15.449 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:01:15.449 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:01:15.449 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:01:15.449 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:01:15.449 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:01:15.449 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:01:15.449 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.449 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.449 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.449 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.449 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:01:15.449 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:01:15.450 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:01:15.450 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:01:15.450 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.450 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.450 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.450 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.450 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:01:15.451 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.451 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.451 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:01:15.451 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_sh_repl": [ 2024-08-19T16:01:15.451 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:01:15.451 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:01:15.451 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "shallow", 2024-08-19T16:01:15.451 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "replicated" 2024-08-19T16:01:15.451 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.451 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:01:15.451 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:01:15.452 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:01:15.452 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:01:15.452 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:01:15.452 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.452 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.452 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.452 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.452 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:01:15.452 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:01:15.452 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.452 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.452 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.453 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.453 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:01:15.453 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:01:15.453 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:01:15.453 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:01:15.453 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:01:15.453 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:01:15.454 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:01:15.454 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.454 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.454 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.454 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.454 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:01:15.454 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:01:15.454 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:01:15.454 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:01:15.454 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.454 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.454 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.455 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.455 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:01:15.455 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.455 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.455 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T16:01:15.455 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:01:15.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T16:01:15.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2024-08-19T16:01:15.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-08-19T16:01:15.541 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T16:01:15.541 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_dp_ec": [ 2024-08-19T16:01:15.541 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:01:15.541 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:01:15.541 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "deep", 2024-08-19T16:01:15.541 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "ec" 2024-08-19T16:01:15.541 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.541 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:01:15.541 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:01:15.542 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:01:15.542 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:01:15.542 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:01:15.542 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.542 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.542 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.542 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.542 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:01:15.542 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:01:15.542 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.542 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.543 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.543 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.543 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:01:15.543 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:01:15.543 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:01:15.543 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:01:15.543 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:01:15.543 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:01:15.543 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:01:15.543 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.544 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.544 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.544 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.544 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:01:15.544 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:01:15.544 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:01:15.544 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:01:15.544 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.544 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.544 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.545 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.545 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:01:15.545 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.545 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.545 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:01:15.545 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_dp_repl": [ 2024-08-19T16:01:15.545 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:01:15.545 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:01:15.545 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "deep", 2024-08-19T16:01:15.545 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "replicated" 2024-08-19T16:01:15.546 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.546 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:01:15.546 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 1, 2024-08-19T16:01:15.546 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 1, 2024-08-19T16:01:15.546 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 1, 2024-08-19T16:01:15.546 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:01:15.546 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 1, 2024-08-19T16:01:15.546 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0.033999302, 2024-08-19T16:01:15.546 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0.033999302 2024-08-19T16:01:15.547 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.547 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:01:15.547 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:01:15.547 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.547 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.547 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.547 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.547 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:01:15.547 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 1, 2024-08-19T16:01:15.547 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:01:15.547 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:01:15.547 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:01:15.547 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:01:15.548 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:01:15.548 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.548 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.548 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.548 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.548 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:01:15.548 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:01:15.548 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 1, 2024-08-19T16:01:15.548 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:01:15.548 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.548 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.549 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.549 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.549 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 2 2024-08-19T16:01:15.549 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.549 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.549 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:01:15.549 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_sh_ec": [ 2024-08-19T16:01:15.549 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:01:15.549 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:01:15.549 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "shallow", 2024-08-19T16:01:15.549 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "ec" 2024-08-19T16:01:15.549 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.549 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:01:15.550 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:01:15.550 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:01:15.550 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:01:15.550 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:01:15.550 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.550 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.550 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.550 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.550 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:01:15.550 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:01:15.551 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.551 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.551 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.551 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.551 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:01:15.551 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:01:15.551 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:01:15.551 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:01:15.551 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:01:15.551 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:01:15.551 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:01:15.552 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.552 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.552 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.552 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.552 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:01:15.552 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:01:15.552 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:01:15.552 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:01:15.552 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.552 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.552 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.552 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.553 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:01:15.553 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.553 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.553 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:01:15.553 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_sh_repl": [ 2024-08-19T16:01:15.553 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:01:15.553 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:01:15.553 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "shallow", 2024-08-19T16:01:15.553 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "replicated" 2024-08-19T16:01:15.553 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.553 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:01:15.553 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 1, 2024-08-19T16:01:15.553 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 1, 2024-08-19T16:01:15.553 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 1, 2024-08-19T16:01:15.554 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:01:15.554 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 1, 2024-08-19T16:01:15.554 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 1.001979462, 2024-08-19T16:01:15.554 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 1.001979462 2024-08-19T16:01:15.554 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.554 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:01:15.554 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:01:15.554 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.554 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.554 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.554 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.554 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:01:15.554 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 1, 2024-08-19T16:01:15.555 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:01:15.555 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:01:15.555 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:01:15.555 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 1, 2024-08-19T16:01:15.555 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:01:15.555 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 1, 2024-08-19T16:01:15.555 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.556 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.556 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.556 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:01:15.556 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:01:15.556 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:01:15.556 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:01:15.556 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.556 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.556 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.556 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.556 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 2 2024-08-19T16:01:15.556 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.557 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.557 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T16:01:15.557 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:01:15.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T16:01:15.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2024-08-19T16:01:15.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-08-19T16:01:15.642 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T16:01:15.642 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_dp_ec": [ 2024-08-19T16:01:15.642 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:01:15.643 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:01:15.643 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "deep", 2024-08-19T16:01:15.643 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "ec" 2024-08-19T16:01:15.643 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.643 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:01:15.643 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:01:15.643 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:01:15.643 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:01:15.643 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:01:15.643 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.643 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.643 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.643 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.644 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:01:15.644 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:01:15.644 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.644 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.644 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.644 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.644 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:01:15.644 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:01:15.644 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:01:15.644 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:01:15.644 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:01:15.645 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:01:15.645 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:01:15.645 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.645 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.645 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.645 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.645 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:01:15.645 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:01:15.645 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:01:15.645 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:01:15.645 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.645 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.645 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.645 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.646 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:01:15.646 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.646 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.646 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:01:15.646 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_dp_repl": [ 2024-08-19T16:01:15.646 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:01:15.646 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:01:15.646 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "deep", 2024-08-19T16:01:15.646 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "replicated" 2024-08-19T16:01:15.646 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.646 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:01:15.646 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:01:15.646 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:01:15.647 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:01:15.647 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:01:15.647 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.647 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.647 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.647 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.647 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:01:15.647 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:01:15.647 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.647 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.647 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.647 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.647 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:01:15.647 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:01:15.648 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:01:15.648 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:01:15.648 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:01:15.648 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:01:15.648 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:01:15.648 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.648 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.648 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.648 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.648 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:01:15.648 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:01:15.648 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:01:15.648 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:01:15.649 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.649 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.649 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.649 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.649 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:01:15.649 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.649 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.649 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:01:15.649 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_sh_ec": [ 2024-08-19T16:01:15.649 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:01:15.649 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:01:15.649 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "shallow", 2024-08-19T16:01:15.649 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "ec" 2024-08-19T16:01:15.650 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.650 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:01:15.650 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:01:15.650 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:01:15.650 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:01:15.650 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:01:15.650 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.650 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.650 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.651 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.651 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:01:15.651 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:01:15.651 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.651 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.651 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.651 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.651 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:01:15.651 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:01:15.651 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:01:15.651 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:01:15.651 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:01:15.652 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:01:15.652 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:01:15.652 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.652 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.652 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.652 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.652 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:01:15.652 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:01:15.652 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:01:15.652 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:01:15.652 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.652 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.653 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.653 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.653 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:01:15.653 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.653 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.653 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:01:15.653 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_sh_repl": [ 2024-08-19T16:01:15.653 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:01:15.653 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:01:15.653 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "shallow", 2024-08-19T16:01:15.653 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "replicated" 2024-08-19T16:01:15.653 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.653 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:01:15.653 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:01:15.654 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:01:15.654 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:01:15.654 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:01:15.654 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.654 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.654 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.654 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.654 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:01:15.654 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:01:15.654 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.654 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.654 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.654 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.655 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:01:15.655 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:01:15.655 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:01:15.655 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:01:15.655 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:01:15.655 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:01:15.655 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:01:15.655 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.656 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.656 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.656 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.656 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:01:15.656 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:01:15.656 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:01:15.656 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:01:15.656 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:01:15.656 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:01:15.656 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:01:15.656 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:01:15.656 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:01:15.656 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.657 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:01:15.657 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T16:01:15.657 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:01:15.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_dump_scrub_schedule ------------------' 2024-08-19T16:01:15.658 INFO:tasks.workunit.client.0.smithi102.stdout:-------------- Teardown Test TEST_dump_scrub_schedule ------------------ 2024-08-19T16:01:15.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2024-08-19T16:01:15.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-08-19T16:01:15.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T16:01:15.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-08-19T16:01:15.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T16:01:15.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T16:01:15.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T16:01:15.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T16:01:15.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T16:01:15.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T16:01:15.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T16:01:15.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T16:01:15.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T16:01:15.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T16:01:15.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T16:01:15.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T16:01:15.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T16:01:15.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T16:01:15.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T16:01:15.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T16:01:15.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T16:01:15.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T16:01:15.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-08-19T16:01:15.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T16:01:15.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:01:15.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:01:15.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.536210 2024-08-19T16:01:15.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T16:01:15.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T16:01:15.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_dump_scrub_schedule ------------------' 2024-08-19T16:01:15.772 INFO:tasks.workunit.client.0.smithi102.stdout:-------------- Complete Test TEST_dump_scrub_schedule ------------------ 2024-08-19T16:01:15.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-08-19T16:01:15.773 INFO:tasks.workunit.client.0.smithi102.stdout:-------------- Prepare Test TEST_interval_changes ------------------- 2024-08-19T16:01:15.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_interval_changes -------------------' 2024-08-19T16:01:15.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-08-19T16:01:15.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-08-19T16:01:15.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-08-19T16:01:15.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-08-19T16:01:15.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T16:01:15.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-08-19T16:01:15.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T16:01:15.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T16:01:15.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T16:01:15.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T16:01:15.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T16:01:15.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T16:01:15.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T16:01:15.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T16:01:15.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T16:01:15.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T16:01:15.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T16:01:15.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T16:01:15.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T16:01:15.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T16:01:15.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T16:01:15.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T16:01:15.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T16:01:15.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T16:01:15.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-08-19T16:01:15.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T16:01:15.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:01:15.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:01:15.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.536210 2024-08-19T16:01:15.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T16:01:15.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T16:01:15.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-08-19T16:01:15.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T16:01:15.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:01:15.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:01:15.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.536210 2024-08-19T16:01:15.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T16:01:15.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T16:01:15.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T16:01:15.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-08-19T16:01:15.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_interval_changes -----------------------' 2024-08-19T16:01:15.785 INFO:tasks.workunit.client.0.smithi102.stdout:-------------- Run Test TEST_interval_changes ----------------------- 2024-08-19T16:01:15.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_interval_changes td/osd-scrub-test 2024-08-19T16:01:15.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:149: TEST_interval_changes: local poolname=test 2024-08-19T16:01:15.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:150: TEST_interval_changes: local OSDS=2 2024-08-19T16:01:15.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:151: TEST_interval_changes: local objects=10 2024-08-19T16:01:15.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:153: TEST_interval_changes: expr 24 '*' 60 '*' 60 2024-08-19T16:01:15.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:153: TEST_interval_changes: local day=86400 2024-08-19T16:01:15.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:154: TEST_interval_changes: expr 86400 '*' 7 2024-08-19T16:01:15.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:154: TEST_interval_changes: local week=604800 2024-08-19T16:01:15.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:155: TEST_interval_changes: local min_interval=86400 2024-08-19T16:01:15.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:156: TEST_interval_changes: local max_interval=604800 2024-08-19T16:01:15.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: local WAIT_FOR_UPDATE=15 2024-08-19T16:01:15.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:159: TEST_interval_changes: TESTDATA=testdata.536210 2024-08-19T16:01:15.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:162: TEST_interval_changes: run_mon td/osd-scrub-test a --osd_pool_default_size=2 2024-08-19T16:01:15.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-test 2024-08-19T16:01:15.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T16:01:15.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T16:01:15.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T16:01:15.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-test/a 2024-08-19T16:01:15.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=2 2024-08-19T16:01:16.166 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T16:01:16.166 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T16:01:16.166 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T16:01:16.167 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T16:01:16.167 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:01:16.167 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:01:16.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T16:01:16.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-08-19T16:01:16.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T16:01:16.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T16:01:16.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T16:01:16.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T16:01:16.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T16:01:16.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T16:01:16.351 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T16:01:16.351 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T16:01:16.352 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T16:01:16.352 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T16:01:16.352 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:01:16.353 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:01:16.353 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.536210/ceph-mon.a.asok 2024-08-19T16:01:16.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T16:01:16.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.536210/ceph-mon.a.asok config get fsid 2024-08-19T16:01:16.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T16:01:16.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T16:01:16.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T16:01:16.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T16:01:16.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T16:01:16.432 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T16:01:16.432 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T16:01:16.432 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T16:01:16.432 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T16:01:16.432 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:01:16.432 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:01:16.432 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.536210/ceph-mon.a.asok 2024-08-19T16:01:16.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T16:01:16.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.536210/ceph-mon.a.asok config get mon_host 2024-08-19T16:01:16.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:163: TEST_interval_changes: run_mgr td/osd-scrub-test x 2024-08-19T16:01:16.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-test 2024-08-19T16:01:16.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T16:01:16.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T16:01:16.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T16:01:16.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-test/x 2024-08-19T16:01:16.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T16:01:17.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T16:01:17.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T16:01:17.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T16:01:17.311 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T16:01:17.311 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:01:17.314 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:01:17.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T16:01:17.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T16:01:17.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T16:01:17.339 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: expr 2 - 1 2024-08-19T16:01:17.340 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: seq 0 1 2024-08-19T16:01:17.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T16:01:17.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:166: TEST_interval_changes: run_osd td/osd-scrub-test 0 --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-08-19T16:01:17.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-08-19T16:01:17.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T16:01:17.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T16:01:17.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T16:01:17.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/0 2024-08-19T16:01:17.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-08-19T16:01:17.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T16:01:17.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T16:01:17.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T16:01:17.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-08-19T16:01:17.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-08-19T16:01:17.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T16:01:17.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T16:01:17.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-08-19T16:01:17.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T16:01:17.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T16:01:17.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T16:01:17.345 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T16:01:17.345 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:01:17.345 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:01:17.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T16:01:17.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T16:01:17.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T16:01:17.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T16:01:17.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T16:01:17.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-08-19T16:01:17.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-08-19T16:01:17.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T16:01:17.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T16:01:17.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T16:01:17.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T16:01:17.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T16:01:17.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0' 2024-08-19T16:01:17.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/0 2024-08-19T16:01:17.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T16:01:17.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=317518cf-695a-4e12-ac7b-59a694209f6b 2024-08-19T16:01:17.347 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 317518cf-695a-4e12-ac7b-59a694209f6b 2024-08-19T16:01:17.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 317518cf-695a-4e12-ac7b-59a694209f6b' 2024-08-19T16:01:17.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T16:01:17.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBNbMNmvpZsFRAAgNZFPw49vzKi5Iqy7pqSUQ== 2024-08-19T16:01:17.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBNbMNmvpZsFRAAgNZFPw49vzKi5Iqy7pqSUQ=="}' 2024-08-19T16:01:17.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 317518cf-695a-4e12-ac7b-59a694209f6b -i td/osd-scrub-test/0/new.json 2024-08-19T16:01:17.558 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T16:01:17.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/0/new.json 2024-08-19T16:01:17.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 --mkfs --key AQBNbMNmvpZsFRAAgNZFPw49vzKi5Iqy7pqSUQ== --osd-uuid 317518cf-695a-4e12-ac7b-59a694209f6b 2024-08-19T16:01:17.597 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:17.596+0000 7f68c4616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:01:17.609 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:17.608+0000 7f68c4616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:01:17.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:17.609+0000 7f68c4616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:01:17.611 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:17.610+0000 7f68c4616640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-08-19T16:01:17.611 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:17.610+0000 7f68c4616640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-08-19T16:01:17.611 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:17.610+0000 7f68c4616640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-08-19T16:01:17.710 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:17.708+0000 7f68c4616640 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-08-19T16:01:23.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-08-19T16:01:23.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T16:01:23.067 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T16:01:23.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T16:01:23.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T16:01:23.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T16:01:23.427 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T16:01:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-08-19T16:01:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T16:01:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T16:01:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T16:01:23.453 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:23.452+0000 7f14948db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:01:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:23.483+0000 7f14948db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:01:23.485 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:23.484+0000 7f14948db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:01:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T16:01:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T16:01:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T16:01:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T16:01:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T16:01:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T16:01:23.693 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T16:01:23.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:23.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:23.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:24.000 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:23.999+0000 7f14948db640 -1 Falling back to public interface 2024-08-19T16:01:24.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:24.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:24.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T16:01:24.954 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T16:01:24.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:24.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:25.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:26.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:26.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:26.225 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T16:01:26.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T16:01:26.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:26.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:26.367 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:26.365+0000 7f14948db640 -1 osd.0 0 log_to_monitors true 2024-08-19T16:01:26.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:27.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:27.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:27.560 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T16:01:27.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T16:01:27.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:27.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:27.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:28.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:28.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:28.816 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T16:01:28.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T16:01:28.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:28.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:29.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:30.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:30.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:30.126 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T16:01:30.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T16:01:30.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:30.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:30.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:31.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:31.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:31.399 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T16:01:31.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T16:01:31.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:31.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:31.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:32.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:32.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:32.674 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T16:01:32.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T16:01:32.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:32.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:32.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:33.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:33.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:33.946 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T16:01:33.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T16:01:33.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:33.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:34.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:35.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:35.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:35.220 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T16:01:35.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T16:01:35.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:35.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:35.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:36.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:36.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:36.490 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T16:01:36.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T16:01:36.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:36.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:36.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:37.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:37.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:37.762 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T16:01:37.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T16:01:37.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:37.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:38.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:39.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:39.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:39.037 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T16:01:39.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T16:01:39.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:39.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:39.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:40.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:40.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:40.656 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T16:01:40.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T16:01:40.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:40.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:41.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:42.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:42.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:42.025 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T16:01:42.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T16:01:42.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:42.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:42.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:43.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:43.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:43.727 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T16:01:43.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T16:01:43.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:43.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:44.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:45.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:45.414 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T16:01:45.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:45.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T16:01:45.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:45.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:45.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:46.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:46.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:46.868 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T16:01:46.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T16:01:46.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:46.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:47.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:48.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:48.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:48.369 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T16:01:48.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T16:01:48.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:48.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:48.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:49.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:49.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:49.646 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T16:01:49.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T16:01:49.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:49.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T16:01:49.917 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3391634800,v1:127.0.0.1:6803/3391634800] [v2:127.0.0.1:6804/3391634800,v1:127.0.0.1:6805/3391634800] exists,up 317518cf-695a-4e12-ac7b-59a694209f6b 2024-08-19T16:01:49.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T16:01:49.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T16:01:49.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T16:01:49.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T16:01:49.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:166: TEST_interval_changes: run_osd td/osd-scrub-test 1 --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-08-19T16:01:49.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-08-19T16:01:49.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T16:01:49.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T16:01:49.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T16:01:49.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/1 2024-08-19T16:01:49.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-08-19T16:01:49.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T16:01:49.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T16:01:49.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T16:01:49.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-08-19T16:01:49.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-08-19T16:01:49.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T16:01:49.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T16:01:49.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-08-19T16:01:49.922 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T16:01:49.922 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T16:01:49.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T16:01:49.923 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T16:01:49.923 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:01:49.924 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:01:49.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T16:01:49.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' 2024-08-19T16:01:49.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T16:01:49.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T16:01:49.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T16:01:49.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-08-19T16:01:49.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-08-19T16:01:49.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T16:01:49.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T16:01:49.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T16:01:49.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T16:01:49.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T16:01:49.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0' 2024-08-19T16:01:49.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/1 2024-08-19T16:01:49.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T16:01:49.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=413e60d7-a37b-4447-aaf9-57e6e6263e9e 2024-08-19T16:01:49.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 413e60d7-a37b-4447-aaf9-57e6e6263e9e' 2024-08-19T16:01:49.925 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 413e60d7-a37b-4447-aaf9-57e6e6263e9e 2024-08-19T16:01:49.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T16:01:49.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBtbMNm/kDMNxAAz2KA1Dyq6QtwtJaOOl7cjA== 2024-08-19T16:01:49.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBtbMNm/kDMNxAAz2KA1Dyq6QtwtJaOOl7cjA=="}' 2024-08-19T16:01:49.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 413e60d7-a37b-4447-aaf9-57e6e6263e9e -i td/osd-scrub-test/1/new.json 2024-08-19T16:01:50.280 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T16:01:50.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/1/new.json 2024-08-19T16:01:50.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 --mkfs --key AQBtbMNm/kDMNxAAz2KA1Dyq6QtwtJaOOl7cjA== --osd-uuid 413e60d7-a37b-4447-aaf9-57e6e6263e9e 2024-08-19T16:01:50.320 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:50.319+0000 7f93bbdcb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:01:50.356 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:50.355+0000 7f93bbdcb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:01:50.357 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:50.356+0000 7f93bbdcb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:01:50.357 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:50.356+0000 7f93bbdcb640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-08-19T16:01:50.357 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:50.356+0000 7f93bbdcb640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-08-19T16:01:50.358 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:50.356+0000 7f93bbdcb640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-08-19T16:01:50.431 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:50.430+0000 7f93bbdcb640 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-08-19T16:01:56.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-08-19T16:01:56.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T16:01:56.317 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T16:01:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T16:01:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T16:01:56.662 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T16:01:56.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T16:01:56.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=275016e2-16bc-4b1b-a8cc-522950720be2 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.536210/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-08-19T16:01:56.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T16:01:56.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T16:01:56.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T16:01:56.689 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:56.687+0000 7fccce219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:01:56.691 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:56.689+0000 7fccce219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:01:56.692 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:56.690+0000 7fccce219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:01:56.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T16:01:56.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T16:01:56.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T16:01:56.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T16:01:56.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T16:01:56.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:56.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T16:01:56.928 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T16:01:56.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:56.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:01:57.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:57.222 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:57.221+0000 7fccce219640 -1 Falling back to public interface 2024-08-19T16:01:58.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:58.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:58.192 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T16:01:58.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T16:01:58.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:58.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:01:58.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:01:59.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:01:59.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:01:59.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T16:01:59.461 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T16:01:59.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:01:59.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:01:59.688 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:01:59.686+0000 7fccce219640 -1 osd.1 0 log_to_monitors true 2024-08-19T16:01:59.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:02:00.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:02:00.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:02:00.790 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T16:02:00.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T16:02:00.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:02:00.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:02:01.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:02:02.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:02:02.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:02:02.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T16:02:02.062 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T16:02:02.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:02:02.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:02:02.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:02:03.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:02:03.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:02:03.338 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T16:02:03.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T16:02:03.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:02:03.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:02:03.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:02:04.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:02:04.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:02:04.700 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T16:02:04.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T16:02:04.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:02:04.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:02:04.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:02:05.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:02:05.973 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T16:02:05.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:02:05.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T16:02:05.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:02:05.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:02:06.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:02:07.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:02:07.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:02:07.324 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T16:02:07.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T16:02:07.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:02:07.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:02:07.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:02:08.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:02:08.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:02:08.770 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T16:02:08.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T16:02:08.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:02:08.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:02:09.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:02:10.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:02:10.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:02:10.045 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T16:02:10.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T16:02:10.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:02:10.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:02:10.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:02:11.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:02:11.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:02:11.323 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T16:02:11.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T16:02:11.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:02:11.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:02:12.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:02:13.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:02:13.075 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T16:02:13.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:02:13.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T16:02:13.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:02:13.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:02:13.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:02:14.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:02:14.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:02:14.573 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T16:02:14.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T16:02:14.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:02:14.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:02:15.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:02:16.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:02:16.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:02:16.230 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T16:02:16.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T16:02:16.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:02:16.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:02:16.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:02:17.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:02:17.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:02:17.579 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T16:02:17.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T16:02:17.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:02:17.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:02:18.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:02:19.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:02:19.340 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T16:02:19.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:02:19.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T16:02:19.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:02:19.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:02:19.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:02:20.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:02:20.768 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T16:02:20.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:02:20.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T16:02:20.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:02:20.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:02:21.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:02:22.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:02:22.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:02:22.495 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T16:02:22.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T16:02:22.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:02:22.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:02:22.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T16:02:23.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T16:02:23.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-08-19T16:02:23.817 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-08-19T16:02:23.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T16:02:23.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T16:02:23.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T16:02:24.119 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1135339615,v1:127.0.0.1:6811/1135339615] [v2:127.0.0.1:6812/1135339615,v1:127.0.0.1:6813/1135339615] exists,up 413e60d7-a37b-4447-aaf9-57e6e6263e9e 2024-08-19T16:02:24.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T16:02:24.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T16:02:24.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T16:02:24.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: create_pool test 1 1 2024-08-19T16:02:24.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-08-19T16:02:24.482 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' created 2024-08-19T16:02:24.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T16:02:25.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:171: TEST_interval_changes: wait_for_clean 2024-08-19T16:02:25.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T16:02:25.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T16:02:25.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T16:02:25.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T16:02:25.496 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T16:02:25.497 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T16:02:25.497 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T16:02:25.497 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T16:02:25.497 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T16:02:25.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T16:02:25.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T16:02:25.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T16:02:25.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T16:02:25.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=500 2024-08-19T16:02:25.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-19T16:02:25.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-19T16:02:25.892 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-08-19T16:02:25.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-19T16:02:25.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T16:02:25.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T16:02:26.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836496 2024-08-19T16:02:26.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836496 2024-08-19T16:02:26.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496' 2024-08-19T16:02:26.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-19T16:02:26.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T16:02:26.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-08-19T16:02:26.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-08-19T16:02:26.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-42949672966' 2024-08-19T16:02:26.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T16:02:26.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836496 2024-08-19T16:02:26.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T16:02:26.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-19T16:02:26.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836496 2024-08-19T16:02:26.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T16:02:26.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836496 2024-08-19T16:02:26.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-08-19T16:02:26.144 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836496 2024-08-19T16:02:26.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T16:02:26.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836496 2024-08-19T16:02:26.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-19T16:02:27.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 500 -eq 0 ']' 2024-08-19T16:02:27.433 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T16:02:27.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836496 2024-08-19T16:02:27.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-19T16:02:27.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-08-19T16:02:27.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-19T16:02:27.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-19T16:02:27.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-08-19T16:02:27.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-19T16:02:27.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-08-19T16:02:27.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-08-19T16:02:27.706 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672966 2024-08-19T16:02:27.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T16:02:28.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672966 2024-08-19T16:02:28.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T16:02:28.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T16:02:28.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T16:02:28.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-19T16:02:28.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T16:02:28.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T16:02:28.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T16:02:28.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T16:02:28.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T16:02:28.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T16:02:28.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T16:02:28.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-19T16:02:28.604 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T16:02:28.605 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T16:02:28.605 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T16:02:28.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-19T16:02:28.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T16:02:28.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T16:02:28.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: ceph osd dump 2024-08-19T16:02:28.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: grep '^pool.*['\'']test['\'']' 2024-08-19T16:02:28.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: awk '{ print $2 }' 2024-08-19T16:02:29.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: local poolid=1 2024-08-19T16:02:29.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:174: TEST_interval_changes: dd if=/dev/urandom of=testdata.536210 bs=1032 count=1 2024-08-19T16:02:29.192 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records in 2024-08-19T16:02:29.192 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records out 2024-08-19T16:02:29.192 INFO:tasks.workunit.client.0.smithi102.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.5571e-05 s, 22.6 MB/s 2024-08-19T16:02:29.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: seq 1 10 2024-08-19T16:02:29.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-08-19T16:02:29.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj1 testdata.536210 2024-08-19T16:02:29.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-08-19T16:02:29.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj2 testdata.536210 2024-08-19T16:02:29.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-08-19T16:02:29.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj3 testdata.536210 2024-08-19T16:02:29.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-08-19T16:02:29.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj4 testdata.536210 2024-08-19T16:02:29.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-08-19T16:02:29.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj5 testdata.536210 2024-08-19T16:02:29.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-08-19T16:02:29.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj6 testdata.536210 2024-08-19T16:02:30.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-08-19T16:02:30.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj7 testdata.536210 2024-08-19T16:02:30.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-08-19T16:02:30.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj8 testdata.536210 2024-08-19T16:02:30.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-08-19T16:02:30.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj9 testdata.536210 2024-08-19T16:02:30.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-08-19T16:02:30.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj10 testdata.536210 2024-08-19T16:02:30.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:179: TEST_interval_changes: rm -f testdata.536210 2024-08-19T16:02:30.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:181: TEST_interval_changes: get_primary test obj1 2024-08-19T16:02:30.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-08-19T16:02:30.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-08-19T16:02:30.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-08-19T16:02:30.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T16:02:30.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:181: TEST_interval_changes: local primary=1 2024-08-19T16:02:30.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:184: TEST_interval_changes: check_dump_scrubs 1 '1 day' '1 week' 2024-08-19T16:02:30.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-08-19T16:02:30.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=1 day' 2024-08-19T16:02:30.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=1 week' 2024-08-19T16:02:30.845 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-08-19T16:02:30.845 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T16:02:30.845 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T16:02:30.846 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T16:02:30.846 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:02:30.847 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:02:30.847 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.536210/ceph-osd.1.asok 2024-08-19T16:02:30.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-08-19T16:02:30.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.536210/ceph-osd.1.asok dump_scrubs 2024-08-19T16:02:30.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-08-19T16:02:30.930 INFO:tasks.workunit.client.0.smithi102.stderr: { 2024-08-19T16:02:30.930 INFO:tasks.workunit.client.0.smithi102.stderr: "pgid": "1.0", 2024-08-19T16:02:30.931 INFO:tasks.workunit.client.0.smithi102.stderr: "sched_time": "2024-08-20T16:02:24.429797+0000", 2024-08-19T16:02:30.931 INFO:tasks.workunit.client.0.smithi102.stderr: "orig_sched_time": "2024-08-20T16:02:24.429797+0000", 2024-08-19T16:02:30.931 INFO:tasks.workunit.client.0.smithi102.stderr: "deadline": "2024-08-26T16:02:24.429797+0000", 2024-08-19T16:02:30.931 INFO:tasks.workunit.client.0.smithi102.stderr: "forced": false 2024-08-19T16:02:30.931 INFO:tasks.workunit.client.0.smithi102.stderr: } 2024-08-19T16:02:30.931 INFO:tasks.workunit.client.0.smithi102.stderr:]' 2024-08-19T16:02:30.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-08-20T16:02:24.429797+0000",' '"orig_sched_time":' '"2024-08-20T16:02:24.429797+0000",' '"deadline":' '"2024-08-26T16:02:24.429797+0000",' '"forced":' false '}' ']' 2024-08-19T16:02:30.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-08-19T16:02:30.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-08-20T16:02:24.429797+0000"' 2024-08-19T16:02:30.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-08-20T16:02:24.429797+0000 2024-08-19T16:02:30.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-08-20T16:02:24.429797+0000 2024-08-19T16:02:30.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-08-19T16:02:30.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 1 day' 2024-08-19T16:02:30.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-08-20 = 2024-08-20 2024-08-19T16:02:30.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-08-20T16:02:24.429797+0000",' '"orig_sched_time":' '"2024-08-20T16:02:24.429797+0000",' '"deadline":' '"2024-08-26T16:02:24.429797+0000",' '"forced":' false '}' ']' 2024-08-19T16:02:30.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-08-19T16:02:30.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-08-26T16:02:24.429797+0000"' 2024-08-19T16:02:30.939 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-08-26T16:02:24.429797+0000 2024-08-19T16:02:30.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-08-26T16:02:24.429797+0000 2024-08-19T16:02:30.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-08-19T16:02:30.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 1 week' 2024-08-19T16:02:30.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-08-26 = 2024-08-26 2024-08-19T16:02:30.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:187: TEST_interval_changes: get_asok_path osd.1 2024-08-19T16:02:30.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T16:02:30.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T16:02:30.944 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T16:02:30.944 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:02:30.944 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:02:30.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.536210/ceph-osd.1.asok 2024-08-19T16:02:30.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:187: TEST_interval_changes: expr 86400 '*' 2 2024-08-19T16:02:30.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:187: TEST_interval_changes: CEPH_ARGS= 2024-08-19T16:02:30.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:187: TEST_interval_changes: ceph --admin-daemon /tmp/ceph-asok.536210/ceph-osd.1.asok config set osd_scrub_min_interval 172800 2024-08-19T16:02:31.022 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T16:02:31.022 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_scrub_min_interval = '' " 2024-08-19T16:02:31.022 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:02:31.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:188: TEST_interval_changes: sleep 15 2024-08-19T16:02:46.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:189: TEST_interval_changes: check_dump_scrubs 1 '2 days' '1 week' 2024-08-19T16:02:46.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-08-19T16:02:46.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=2 days' 2024-08-19T16:02:46.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=1 week' 2024-08-19T16:02:46.032 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-08-19T16:02:46.032 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T16:02:46.032 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T16:02:46.032 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T16:02:46.032 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:02:46.032 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:02:46.032 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.536210/ceph-osd.1.asok 2024-08-19T16:02:46.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-08-19T16:02:46.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.536210/ceph-osd.1.asok dump_scrubs 2024-08-19T16:02:46.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-08-19T16:02:46.126 INFO:tasks.workunit.client.0.smithi102.stderr: { 2024-08-19T16:02:46.127 INFO:tasks.workunit.client.0.smithi102.stderr: "pgid": "1.0", 2024-08-19T16:02:46.127 INFO:tasks.workunit.client.0.smithi102.stderr: "sched_time": "2024-08-21T16:02:24.429797+0000", 2024-08-19T16:02:46.127 INFO:tasks.workunit.client.0.smithi102.stderr: "orig_sched_time": "2024-08-21T16:02:24.429797+0000", 2024-08-19T16:02:46.127 INFO:tasks.workunit.client.0.smithi102.stderr: "deadline": "2024-08-26T16:02:24.429797+0000", 2024-08-19T16:02:46.127 INFO:tasks.workunit.client.0.smithi102.stderr: "forced": false 2024-08-19T16:02:46.127 INFO:tasks.workunit.client.0.smithi102.stderr: } 2024-08-19T16:02:46.127 INFO:tasks.workunit.client.0.smithi102.stderr:]' 2024-08-19T16:02:46.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-08-21T16:02:24.429797+0000",' '"orig_sched_time":' '"2024-08-21T16:02:24.429797+0000",' '"deadline":' '"2024-08-26T16:02:24.429797+0000",' '"forced":' false '}' ']' 2024-08-19T16:02:46.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-08-19T16:02:46.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-08-21T16:02:24.429797+0000"' 2024-08-19T16:02:46.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-08-21T16:02:24.429797+0000 2024-08-19T16:02:46.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-08-21T16:02:24.429797+0000 2024-08-19T16:02:46.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-08-19T16:02:46.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 days' 2024-08-19T16:02:46.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-08-21 = 2024-08-21 2024-08-19T16:02:46.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-08-21T16:02:24.429797+0000",' '"orig_sched_time":' '"2024-08-21T16:02:24.429797+0000",' '"deadline":' '"2024-08-26T16:02:24.429797+0000",' '"forced":' false '}' ']' 2024-08-19T16:02:46.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-08-19T16:02:46.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-08-26T16:02:24.429797+0000"' 2024-08-19T16:02:46.136 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-08-26T16:02:24.429797+0000 2024-08-19T16:02:46.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-08-26T16:02:24.429797+0000 2024-08-19T16:02:46.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-08-19T16:02:46.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 1 week' 2024-08-19T16:02:46.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-08-26 = 2024-08-26 2024-08-19T16:02:46.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:192: TEST_interval_changes: get_asok_path osd.1 2024-08-19T16:02:46.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T16:02:46.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T16:02:46.140 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T16:02:46.140 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:02:46.140 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:02:46.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.536210/ceph-osd.1.asok 2024-08-19T16:02:46.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:192: TEST_interval_changes: expr 604800 '*' 2 2024-08-19T16:02:46.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:192: TEST_interval_changes: CEPH_ARGS= 2024-08-19T16:02:46.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:192: TEST_interval_changes: ceph --admin-daemon /tmp/ceph-asok.536210/ceph-osd.1.asok config set osd_scrub_max_interval 1209600 2024-08-19T16:02:46.220 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T16:02:46.220 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "osd_scrub_max_interval = '' " 2024-08-19T16:02:46.220 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:02:46.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:193: TEST_interval_changes: sleep 15 2024-08-19T16:03:01.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:194: TEST_interval_changes: check_dump_scrubs 1 '2 days' '2 week' 2024-08-19T16:03:01.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-08-19T16:03:01.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=2 days' 2024-08-19T16:03:01.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=2 week' 2024-08-19T16:03:01.231 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-08-19T16:03:01.231 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T16:03:01.231 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T16:03:01.231 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T16:03:01.231 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:03:01.231 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:03:01.231 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.536210/ceph-osd.1.asok 2024-08-19T16:03:01.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-08-19T16:03:01.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.536210/ceph-osd.1.asok dump_scrubs 2024-08-19T16:03:01.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-08-19T16:03:01.323 INFO:tasks.workunit.client.0.smithi102.stderr: { 2024-08-19T16:03:01.323 INFO:tasks.workunit.client.0.smithi102.stderr: "pgid": "1.0", 2024-08-19T16:03:01.323 INFO:tasks.workunit.client.0.smithi102.stderr: "sched_time": "2024-08-21T16:02:24.429797+0000", 2024-08-19T16:03:01.323 INFO:tasks.workunit.client.0.smithi102.stderr: "orig_sched_time": "2024-08-21T16:02:24.429797+0000", 2024-08-19T16:03:01.323 INFO:tasks.workunit.client.0.smithi102.stderr: "deadline": "2024-09-02T16:02:24.429797+0000", 2024-08-19T16:03:01.324 INFO:tasks.workunit.client.0.smithi102.stderr: "forced": false 2024-08-19T16:03:01.324 INFO:tasks.workunit.client.0.smithi102.stderr: } 2024-08-19T16:03:01.324 INFO:tasks.workunit.client.0.smithi102.stderr:]' 2024-08-19T16:03:01.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-08-21T16:02:24.429797+0000",' '"orig_sched_time":' '"2024-08-21T16:02:24.429797+0000",' '"deadline":' '"2024-09-02T16:02:24.429797+0000",' '"forced":' false '}' ']' 2024-08-19T16:03:01.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-08-19T16:03:01.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-08-21T16:02:24.429797+0000"' 2024-08-19T16:03:01.326 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-08-21T16:02:24.429797+0000 2024-08-19T16:03:01.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-08-21T16:02:24.429797+0000 2024-08-19T16:03:01.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-08-19T16:03:01.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 days' 2024-08-19T16:03:01.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-08-21 = 2024-08-21 2024-08-19T16:03:01.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-08-21T16:02:24.429797+0000",' '"orig_sched_time":' '"2024-08-21T16:02:24.429797+0000",' '"deadline":' '"2024-09-02T16:02:24.429797+0000",' '"forced":' false '}' ']' 2024-08-19T16:03:01.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-08-19T16:03:01.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-09-02T16:02:24.429797+0000"' 2024-08-19T16:03:01.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-09-02T16:02:24.429797+0000 2024-08-19T16:03:01.333 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-09-02T16:02:24.429797+0000 2024-08-19T16:03:01.333 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-08-19T16:03:01.334 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 week' 2024-08-19T16:03:01.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-09-02 = 2024-09-02 2024-08-19T16:03:01.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:197: TEST_interval_changes: expr 86400 '*' 3 2024-08-19T16:03:01.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:197: TEST_interval_changes: ceph osd pool set test scrub_min_interval 259200 2024-08-19T16:03:01.729 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 scrub_min_interval to 259200 2024-08-19T16:03:01.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:198: TEST_interval_changes: sleep 15 2024-08-19T16:03:16.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:199: TEST_interval_changes: check_dump_scrubs 1 '3 days' '2 week' 2024-08-19T16:03:16.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-08-19T16:03:16.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=3 days' 2024-08-19T16:03:16.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=2 week' 2024-08-19T16:03:16.744 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-08-19T16:03:16.744 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T16:03:16.744 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T16:03:16.745 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T16:03:16.745 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:03:16.745 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:03:16.745 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.536210/ceph-osd.1.asok 2024-08-19T16:03:16.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-08-19T16:03:16.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.536210/ceph-osd.1.asok dump_scrubs 2024-08-19T16:03:16.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-08-19T16:03:16.836 INFO:tasks.workunit.client.0.smithi102.stderr: { 2024-08-19T16:03:16.836 INFO:tasks.workunit.client.0.smithi102.stderr: "pgid": "1.0", 2024-08-19T16:03:16.836 INFO:tasks.workunit.client.0.smithi102.stderr: "sched_time": "2024-08-22T16:02:24.429797+0000", 2024-08-19T16:03:16.836 INFO:tasks.workunit.client.0.smithi102.stderr: "orig_sched_time": "2024-08-22T16:02:24.429797+0000", 2024-08-19T16:03:16.836 INFO:tasks.workunit.client.0.smithi102.stderr: "deadline": "2024-09-02T16:02:24.429797+0000", 2024-08-19T16:03:16.836 INFO:tasks.workunit.client.0.smithi102.stderr: "forced": false 2024-08-19T16:03:16.836 INFO:tasks.workunit.client.0.smithi102.stderr: } 2024-08-19T16:03:16.836 INFO:tasks.workunit.client.0.smithi102.stderr:]' 2024-08-19T16:03:16.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-08-22T16:02:24.429797+0000",' '"orig_sched_time":' '"2024-08-22T16:02:24.429797+0000",' '"deadline":' '"2024-09-02T16:02:24.429797+0000",' '"forced":' false '}' ']' 2024-08-19T16:03:16.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-08-19T16:03:16.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-08-22T16:02:24.429797+0000"' 2024-08-19T16:03:16.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-08-22T16:02:24.429797+0000 2024-08-19T16:03:16.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-08-22T16:02:24.429797+0000 2024-08-19T16:03:16.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-08-19T16:03:16.841 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 days' 2024-08-19T16:03:16.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-08-22 = 2024-08-22 2024-08-19T16:03:16.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-08-22T16:02:24.429797+0000",' '"orig_sched_time":' '"2024-08-22T16:02:24.429797+0000",' '"deadline":' '"2024-09-02T16:02:24.429797+0000",' '"forced":' false '}' ']' 2024-08-19T16:03:16.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-08-19T16:03:16.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-09-02T16:02:24.429797+0000"' 2024-08-19T16:03:16.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-09-02T16:02:24.429797+0000 2024-08-19T16:03:16.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-09-02T16:02:24.429797+0000 2024-08-19T16:03:16.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-08-19T16:03:16.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 week' 2024-08-19T16:03:16.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-09-02 = 2024-09-02 2024-08-19T16:03:16.849 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:202: TEST_interval_changes: expr 604800 '*' 3 2024-08-19T16:03:16.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:202: TEST_interval_changes: ceph osd pool set test scrub_max_interval 1814400 2024-08-19T16:03:17.210 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 scrub_max_interval to 1814400 2024-08-19T16:03:17.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:203: TEST_interval_changes: sleep 15 2024-08-19T16:03:32.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:204: TEST_interval_changes: check_dump_scrubs 1 '3 days' '3 week' 2024-08-19T16:03:32.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-08-19T16:03:32.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=3 days' 2024-08-19T16:03:32.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=3 week' 2024-08-19T16:03:32.225 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-08-19T16:03:32.225 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T16:03:32.225 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T16:03:32.225 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T16:03:32.225 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:03:32.225 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:03:32.225 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.536210/ceph-osd.1.asok 2024-08-19T16:03:32.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-08-19T16:03:32.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.536210/ceph-osd.1.asok dump_scrubs 2024-08-19T16:03:32.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-08-19T16:03:32.322 INFO:tasks.workunit.client.0.smithi102.stderr: { 2024-08-19T16:03:32.322 INFO:tasks.workunit.client.0.smithi102.stderr: "pgid": "1.0", 2024-08-19T16:03:32.322 INFO:tasks.workunit.client.0.smithi102.stderr: "sched_time": "2024-08-22T16:02:24.429797+0000", 2024-08-19T16:03:32.322 INFO:tasks.workunit.client.0.smithi102.stderr: "orig_sched_time": "2024-08-22T16:02:24.429797+0000", 2024-08-19T16:03:32.323 INFO:tasks.workunit.client.0.smithi102.stderr: "deadline": "2024-09-09T16:02:24.429797+0000", 2024-08-19T16:03:32.323 INFO:tasks.workunit.client.0.smithi102.stderr: "forced": false 2024-08-19T16:03:32.323 INFO:tasks.workunit.client.0.smithi102.stderr: } 2024-08-19T16:03:32.323 INFO:tasks.workunit.client.0.smithi102.stderr:]' 2024-08-19T16:03:32.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-08-22T16:02:24.429797+0000",' '"orig_sched_time":' '"2024-08-22T16:02:24.429797+0000",' '"deadline":' '"2024-09-09T16:02:24.429797+0000",' '"forced":' false '}' ']' 2024-08-19T16:03:32.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-08-19T16:03:32.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-08-22T16:02:24.429797+0000"' 2024-08-19T16:03:32.326 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-08-22T16:02:24.429797+0000 2024-08-19T16:03:32.326 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-08-22T16:02:24.429797+0000 2024-08-19T16:03:32.326 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-08-19T16:03:32.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 days' 2024-08-19T16:03:32.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-08-22 = 2024-08-22 2024-08-19T16:03:32.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-08-22T16:02:24.429797+0000",' '"orig_sched_time":' '"2024-08-22T16:02:24.429797+0000",' '"deadline":' '"2024-09-09T16:02:24.429797+0000",' '"forced":' false '}' ']' 2024-08-19T16:03:32.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-08-19T16:03:32.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-09-09T16:02:24.429797+0000"' 2024-08-19T16:03:32.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-09-09T16:02:24.429797+0000 2024-08-19T16:03:32.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-09-09T16:02:24.429797+0000 2024-08-19T16:03:32.333 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-08-19T16:03:32.334 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 week' 2024-08-19T16:03:32.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-09-09 = 2024-09-09 2024-08-19T16:03:32.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:205: TEST_interval_changes: perf_counters td/osd-scrub-test 2 2024-08-19T16:03:32.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2024-08-19T16:03:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=2 2024-08-19T16:03:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 2 - 1 2024-08-19T16:03:32.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 1 2024-08-19T16:03:32.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T16:03:32.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2024-08-19T16:03:32.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-08-19T16:03:32.433 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T16:03:32.434 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_dp_ec": [ 2024-08-19T16:03:32.434 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:03:32.434 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:03:32.434 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "deep", 2024-08-19T16:03:32.434 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "ec" 2024-08-19T16:03:32.434 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.434 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:03:32.434 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:03:32.434 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:03:32.434 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:03:32.434 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:03:32.435 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.435 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.435 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.435 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.435 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:03:32.435 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:03:32.435 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.435 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.435 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.435 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.435 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:03:32.435 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:03:32.436 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:03:32.436 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:03:32.436 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:03:32.436 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:03:32.437 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:03:32.437 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.437 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.437 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.437 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.438 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:03:32.438 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:03:32.438 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:03:32.438 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:03:32.438 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.438 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.438 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.439 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.439 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:03:32.439 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:03:32.439 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:03:32.439 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:03:32.439 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_dp_repl": [ 2024-08-19T16:03:32.439 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:03:32.439 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:03:32.439 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "deep", 2024-08-19T16:03:32.439 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "replicated" 2024-08-19T16:03:32.439 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.439 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:03:32.439 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:03:32.440 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:03:32.440 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:03:32.440 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:03:32.440 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.440 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.440 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.440 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.440 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:03:32.440 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:03:32.440 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.440 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.440 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.441 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.441 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:03:32.441 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:03:32.441 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:03:32.441 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:03:32.441 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:03:32.441 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:03:32.441 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:03:32.441 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.441 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.441 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.441 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.442 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:03:32.442 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:03:32.442 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:03:32.442 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:03:32.442 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.442 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.442 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.442 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.442 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:03:32.442 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:03:32.442 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:03:32.442 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:03:32.443 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_sh_ec": [ 2024-08-19T16:03:32.443 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:03:32.443 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:03:32.443 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "shallow", 2024-08-19T16:03:32.443 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "ec" 2024-08-19T16:03:32.443 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.443 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:03:32.443 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:03:32.443 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:03:32.443 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:03:32.443 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:03:32.443 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.443 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.443 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.444 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.444 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:03:32.444 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:03:32.444 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.444 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.444 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.444 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.444 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:03:32.444 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:03:32.444 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:03:32.444 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:03:32.444 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:03:32.444 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:03:32.445 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:03:32.445 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.445 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.445 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.445 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.445 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:03:32.445 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:03:32.445 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:03:32.445 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:03:32.445 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.445 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.445 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.445 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.446 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:03:32.446 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:03:32.446 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:03:32.446 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:03:32.446 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_sh_repl": [ 2024-08-19T16:03:32.446 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:03:32.446 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:03:32.446 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "shallow", 2024-08-19T16:03:32.446 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "replicated" 2024-08-19T16:03:32.446 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.446 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:03:32.446 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:03:32.447 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:03:32.447 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:03:32.447 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:03:32.447 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.447 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.447 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.447 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.447 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:03:32.447 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:03:32.447 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.447 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.447 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.447 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.447 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:03:32.448 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:03:32.448 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:03:32.448 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:03:32.448 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:03:32.448 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:03:32.448 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:03:32.448 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.449 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.449 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.449 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.449 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:03:32.449 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:03:32.449 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:03:32.449 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:03:32.449 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.449 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.449 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.449 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.449 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:03:32.449 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:03:32.450 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:03:32.450 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T16:03:32.450 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:03:32.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-08-19T16:03:32.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2024-08-19T16:03:32.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-08-19T16:03:32.535 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T16:03:32.535 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_dp_ec": [ 2024-08-19T16:03:32.535 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:03:32.535 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:03:32.535 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "deep", 2024-08-19T16:03:32.535 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "ec" 2024-08-19T16:03:32.535 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.535 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:03:32.535 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:03:32.536 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:03:32.536 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:03:32.536 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:03:32.536 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.536 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.536 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.536 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.536 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:03:32.536 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:03:32.536 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.536 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.536 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.536 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.537 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:03:32.537 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:03:32.537 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:03:32.537 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:03:32.537 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:03:32.537 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:03:32.537 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:03:32.537 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.537 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.537 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.537 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.537 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:03:32.537 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:03:32.538 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:03:32.538 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:03:32.538 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.538 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.538 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.538 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.538 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:03:32.538 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:03:32.538 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:03:32.538 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:03:32.538 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_dp_repl": [ 2024-08-19T16:03:32.538 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:03:32.538 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:03:32.539 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "deep", 2024-08-19T16:03:32.539 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "replicated" 2024-08-19T16:03:32.539 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.539 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:03:32.539 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:03:32.539 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:03:32.539 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:03:32.539 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:03:32.539 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.539 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.539 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.539 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.539 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:03:32.539 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:03:32.540 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.540 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.540 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.540 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.540 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:03:32.540 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:03:32.540 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:03:32.540 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:03:32.540 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:03:32.540 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:03:32.540 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:03:32.540 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.540 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.541 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.541 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.541 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:03:32.541 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:03:32.541 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:03:32.541 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:03:32.541 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.541 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.541 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.541 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.541 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:03:32.541 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:03:32.541 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:03:32.541 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:03:32.542 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_sh_ec": [ 2024-08-19T16:03:32.542 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:03:32.542 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:03:32.542 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "shallow", 2024-08-19T16:03:32.542 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "ec" 2024-08-19T16:03:32.542 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.542 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:03:32.542 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:03:32.542 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:03:32.542 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:03:32.542 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:03:32.542 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.543 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.543 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.543 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.543 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:03:32.543 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:03:32.543 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.543 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.543 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.543 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.543 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:03:32.543 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:03:32.543 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:03:32.543 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:03:32.544 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:03:32.544 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:03:32.544 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:03:32.544 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.544 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.544 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.544 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.544 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:03:32.544 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:03:32.544 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:03:32.544 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:03:32.544 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.544 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.544 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.545 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.545 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:03:32.545 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:03:32.545 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:03:32.545 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-08-19T16:03:32.545 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_scrub_sh_repl": [ 2024-08-19T16:03:32.545 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-08-19T16:03:32.545 INFO:tasks.workunit.client.0.smithi102.stdout: "labels": { 2024-08-19T16:03:32.545 INFO:tasks.workunit.client.0.smithi102.stdout: "level": "shallow", 2024-08-19T16:03:32.545 INFO:tasks.workunit.client.0.smithi102.stdout: "pooltype": "replicated" 2024-08-19T16:03:32.545 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.545 INFO:tasks.workunit.client.0.smithi102.stdout: "counters": { 2024-08-19T16:03:32.545 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_started": 0, 2024-08-19T16:03:32.546 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrubs_past_reservation": 0, 2024-08-19T16:03:32.546 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs": 0, 2024-08-19T16:03:32.546 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_scrubs_elapsed": { 2024-08-19T16:03:32.546 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.546 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.546 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.546 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.546 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs": 0, 2024-08-19T16:03:32.546 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_scrubs_elapsed": { 2024-08-19T16:03:32.546 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.546 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.546 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.546 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.546 INFO:tasks.workunit.client.0.smithi102.stdout: "preemptions": 0, 2024-08-19T16:03:32.547 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_selected": 0, 2024-08-19T16:03:32.547 INFO:tasks.workunit.client.0.smithi102.stdout: "chunk_busy": 0, 2024-08-19T16:03:32.547 INFO:tasks.workunit.client.0.smithi102.stdout: "locked_object": 0, 2024-08-19T16:03:32.547 INFO:tasks.workunit.client.0.smithi102.stdout: "write_blocked_by_scrub": 0, 2024-08-19T16:03:32.547 INFO:tasks.workunit.client.0.smithi102.stdout: "scrub_reservations_completed": 0, 2024-08-19T16:03:32.547 INFO:tasks.workunit.client.0.smithi102.stdout: "successful_reservations_elapsed": { 2024-08-19T16:03:32.547 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.547 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.547 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.548 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.548 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_aborted": 0, 2024-08-19T16:03:32.548 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_failure": 0, 2024-08-19T16:03:32.548 INFO:tasks.workunit.client.0.smithi102.stdout: "reservation_process_skipped": 0, 2024-08-19T16:03:32.548 INFO:tasks.workunit.client.0.smithi102.stdout: "failed_reservations_elapsed": { 2024-08-19T16:03:32.548 INFO:tasks.workunit.client.0.smithi102.stdout: "avgcount": 0, 2024-08-19T16:03:32.548 INFO:tasks.workunit.client.0.smithi102.stdout: "sum": 0E-9, 2024-08-19T16:03:32.548 INFO:tasks.workunit.client.0.smithi102.stdout: "avgtime": 0E-9 2024-08-19T16:03:32.548 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-08-19T16:03:32.548 INFO:tasks.workunit.client.0.smithi102.stdout: "replicas_in_reservation": 0 2024-08-19T16:03:32.548 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:03:32.548 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-08-19T16:03:32.548 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-08-19T16:03:32.549 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:03:32.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_interval_changes ------------------' 2024-08-19T16:03:32.549 INFO:tasks.workunit.client.0.smithi102.stdout:-------------- Teardown Test TEST_interval_changes ------------------ 2024-08-19T16:03:32.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2024-08-19T16:03:32.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-08-19T16:03:32.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T16:03:32.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-08-19T16:03:32.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T16:03:32.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T16:03:32.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T16:03:32.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T16:03:32.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T16:03:32.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T16:03:32.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T16:03:32.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T16:03:32.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T16:03:32.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T16:03:32.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T16:03:32.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T16:03:32.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T16:03:32.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T16:03:32.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T16:03:32.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T16:03:32.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T16:03:32.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T16:03:32.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-08-19T16:03:32.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T16:03:32.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:03:32.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:03:32.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.536210 2024-08-19T16:03:32.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T16:03:32.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T16:03:32.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_interval_changes ------------------' 2024-08-19T16:03:32.667 INFO:tasks.workunit.client.0.smithi102.stdout:-------------- Complete Test TEST_interval_changes ------------------ 2024-08-19T16:03:32.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-08-19T16:03:32.668 INFO:tasks.workunit.client.0.smithi102.stdout:-------------- Prepare Test TEST_just_deep_scrubs ------------------- 2024-08-19T16:03:32.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_just_deep_scrubs -------------------' 2024-08-19T16:03:32.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-08-19T16:03:32.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-08-19T16:03:32.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-08-19T16:03:32.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-08-19T16:03:32.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T16:03:32.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-08-19T16:03:32.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T16:03:32.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T16:03:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T16:03:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T16:03:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T16:03:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T16:03:32.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T16:03:32.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T16:03:32.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T16:03:32.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T16:03:32.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T16:03:32.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T16:03:32.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T16:03:32.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T16:03:32.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T16:03:32.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T16:03:32.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T16:03:32.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T16:03:32.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-08-19T16:03:32.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T16:03:32.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:03:32.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:03:32.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.536210 2024-08-19T16:03:32.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T16:03:32.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T16:03:32.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-08-19T16:03:32.678 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T16:03:32.678 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:03:32.678 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:03:32.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.536210 2024-08-19T16:03:32.679 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T16:03:32.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T16:03:32.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T16:03:32.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-08-19T16:03:32.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_just_deep_scrubs -----------------------' 2024-08-19T16:03:32.680 INFO:tasks.workunit.client.0.smithi102.stdout:-------------- Run Test TEST_just_deep_scrubs ----------------------- 2024-08-19T16:03:32.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_just_deep_scrubs td/osd-scrub-test 2024-08-19T16:03:32.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:467: TEST_just_deep_scrubs: local dir=td/osd-scrub-test 2024-08-19T16:03:32.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:468: TEST_just_deep_scrubs: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-08-19T16:03:32.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:468: TEST_just_deep_scrubs: local -A cluster_conf 2024-08-19T16:03:32.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:474: TEST_just_deep_scrubs: standard_scrub_cluster td/osd-scrub-test cluster_conf 2024-08-19T16:03:32.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-scrub-test 2024-08-19T16:03:32.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-08-19T16:03:32.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-08-19T16:03:32.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2024-08-19T16:03:32.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2024-08-19T16:03:32.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2024-08-19T16:03:32.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2024-08-19T16:03:32.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-08-19T16:03:32.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-08-19T16:03:32.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-08-19T16:03:34.244 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 2ed5401c-3d47-4a8e-ba39-978510e3a54f 2024-08-19T16:03:34.460 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T16:03:34.502 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:03:34.500+0000 7ff6f9101640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:03:34.511 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:03:34.509+0000 7ff6f9101640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:03:34.512 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:03:34.511+0000 7ff6f9101640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:03:34.513 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:03:34.511+0000 7ff6f9101640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-08-19T16:03:34.513 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:03:34.511+0000 7ff6f9101640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-08-19T16:03:34.513 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:03:34.511+0000 7ff6f9101640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-08-19T16:03:34.594 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:03:34.593+0000 7ff6f9101640 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-08-19T16:03:40.033 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-08-19T16:03:40.379 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-08-19T16:03:40.408 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:03:40.407+0000 7fa1c0486640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:03:40.410 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:03:40.409+0000 7fa1c0486640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:03:40.411 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:03:40.410+0000 7fa1c0486640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:03:40.646 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T16:03:40.931 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:03:40.930+0000 7fa1c0486640 -1 Falling back to public interface 2024-08-19T16:03:41.911 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T16:03:43.180 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T16:03:43.235 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:03:43.233+0000 7fa1c0486640 -1 osd.0 0 log_to_monitors true 2024-08-19T16:03:44.470 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T16:03:45.745 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T16:03:47.136 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T16:03:48.408 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T16:03:49.681 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T16:03:50.957 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T16:03:52.230 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T16:03:53.504 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T16:03:54.789 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T16:03:56.394 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T16:03:57.962 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T16:03:59.703 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T16:04:01.263 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T16:04:03.008 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T16:04:04.719 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T16:04:06.064 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T16:04:06.342 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3186403487,v1:127.0.0.1:6803/3186403487] [v2:127.0.0.1:6804/3186403487,v1:127.0.0.1:6805/3186403487] exists,up 2ed5401c-3d47-4a8e-ba39-978510e3a54f 2024-08-19T16:04:06.346 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 cd732cf3-e48d-4cb4-8df2-35af41538ca0 2024-08-19T16:04:06.707 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T16:04:06.749 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:06.747+0000 7f0094e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:04:06.783 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:06.781+0000 7f0094e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:04:06.784 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:06.782+0000 7f0094e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:04:06.784 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:06.782+0000 7f0094e14640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-08-19T16:04:06.785 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:06.782+0000 7f0094e14640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-08-19T16:04:06.785 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:06.782+0000 7f0094e14640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-08-19T16:04:06.858 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:06.856+0000 7f0094e14640 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-08-19T16:04:12.284 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-08-19T16:04:12.625 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-08-19T16:04:12.654 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:12.652+0000 7f0759101640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:04:12.656 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:12.654+0000 7f0759101640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:04:12.657 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:12.655+0000 7f0759101640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:04:12.889 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T16:04:13.181 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:13.180+0000 7f0759101640 -1 Falling back to public interface 2024-08-19T16:04:14.155 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T16:04:15.438 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T16:04:15.540 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:15.538+0000 7f0759101640 -1 osd.1 0 log_to_monitors true 2024-08-19T16:04:16.732 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T16:04:18.159 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T16:04:19.583 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T16:04:20.856 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T16:04:22.174 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T16:04:23.625 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T16:04:24.896 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T16:04:26.169 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T16:04:27.644 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T16:04:28.907 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T16:04:30.310 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T16:04:31.754 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T16:04:33.490 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T16:04:35.218 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T16:04:36.645 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T16:04:38.531 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T16:04:38.848 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/372694342,v1:127.0.0.1:6811/372694342] [v2:127.0.0.1:6812/372694342,v1:127.0.0.1:6813/372694342] exists,up cd732cf3-e48d-4cb4-8df2-35af41538ca0 2024-08-19T16:04:38.852 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 0096e214-dc03-4138-a73f-0a585ce6e1f0 2024-08-19T16:04:39.191 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T16:04:39.234 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:39.232+0000 7fa681725640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:04:39.267 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:39.266+0000 7fa681725640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:04:39.268 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:39.267+0000 7fa681725640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:04:39.269 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:39.267+0000 7fa681725640 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-08-19T16:04:39.269 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:39.267+0000 7fa681725640 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-08-19T16:04:39.269 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:39.267+0000 7fa681725640 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-08-19T16:04:39.343 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:39.341+0000 7fa681725640 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-08-19T16:04:45.367 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-08-19T16:04:45.707 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-08-19T16:04:45.736 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:45.735+0000 7fc4fa588640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:04:45.738 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:45.737+0000 7fc4fa588640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:04:45.739 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:45.738+0000 7fc4fa588640 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T16:04:45.973 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-08-19T16:04:46.269 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:46.268+0000 7fc4fa588640 -1 Falling back to public interface 2024-08-19T16:04:47.243 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-08-19T16:04:48.513 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-08-19T16:04:48.750 INFO:tasks.workunit.client.0.smithi102.stderr:2024-08-19T16:04:48.748+0000 7fc4fa588640 -1 osd.2 0 log_to_monitors true 2024-08-19T16:04:49.788 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-08-19T16:04:51.063 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-08-19T16:04:52.342 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-08-19T16:04:53.725 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-08-19T16:04:55.002 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-08-19T16:04:56.336 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-08-19T16:04:57.611 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-08-19T16:04:58.885 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-08-19T16:05:00.193 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-08-19T16:05:01.758 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-08-19T16:05:03.310 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-08-19T16:05:04.963 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-08-19T16:05:06.506 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-08-19T16:05:07.940 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-08-19T16:05:09.375 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-08-19T16:05:10.957 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-08-19T16:05:11.229 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2087475914,v1:127.0.0.1:6819/2087475914] [v2:127.0.0.1:6820/2087475914,v1:127.0.0.1:6821/2087475914] exists,up 0096e214-dc03-4138-a73f-0a585ce6e1f0 2024-08-19T16:05:11.639 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' created 2024-08-19T16:05:13.337 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836497 2024-08-19T16:05:14.873 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672971 2024-08-19T16:05:15.142 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509446 2024-08-19T16:05:17.611 INFO:tasks.workunit.client.0.smithi102.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-08-19T16:05:17.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:475: TEST_just_deep_scrubs: local poolid=1 2024-08-19T16:05:17.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:476: TEST_just_deep_scrubs: local poolname=test 2024-08-19T16:05:17.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: echo 'Pool: test : 1' 2024-08-19T16:05:17.612 INFO:tasks.workunit.client.0.smithi102.stdout:Pool: test : 1 2024-08-19T16:05:17.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: TESTDATA=testdata.536210 2024-08-19T16:05:17.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:480: TEST_just_deep_scrubs: local objects=15 2024-08-19T16:05:17.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:481: TEST_just_deep_scrubs: dd if=/dev/urandom of=testdata.536210 bs=1032 count=1 2024-08-19T16:05:17.613 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records in 2024-08-19T16:05:17.613 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records out 2024-08-19T16:05:17.613 INFO:tasks.workunit.client.0.smithi102.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.6971e-05 s, 22.0 MB/s 2024-08-19T16:05:17.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: seq 1 15 2024-08-19T16:05:17.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-08-19T16:05:17.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj1 testdata.536210 2024-08-19T16:05:17.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-08-19T16:05:17.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj2 testdata.536210 2024-08-19T16:05:17.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-08-19T16:05:17.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj3 testdata.536210 2024-08-19T16:05:18.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-08-19T16:05:18.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj4 testdata.536210 2024-08-19T16:05:18.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-08-19T16:05:18.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj5 testdata.536210 2024-08-19T16:05:18.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-08-19T16:05:18.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj6 testdata.536210 2024-08-19T16:05:18.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-08-19T16:05:18.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj7 testdata.536210 2024-08-19T16:05:18.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-08-19T16:05:18.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj8 testdata.536210 2024-08-19T16:05:18.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-08-19T16:05:18.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj9 testdata.536210 2024-08-19T16:05:18.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-08-19T16:05:18.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj10 testdata.536210 2024-08-19T16:05:19.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-08-19T16:05:19.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj11 testdata.536210 2024-08-19T16:05:19.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-08-19T16:05:19.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj12 testdata.536210 2024-08-19T16:05:19.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-08-19T16:05:19.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj13 testdata.536210 2024-08-19T16:05:19.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-08-19T16:05:19.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj14 testdata.536210 2024-08-19T16:05:19.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-08-19T16:05:19.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj15 testdata.536210 2024-08-19T16:05:19.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: rm -f testdata.536210 2024-08-19T16:05:19.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:491: TEST_just_deep_scrubs: ceph osd set noscrub 2024-08-19T16:05:20.053 INFO:tasks.workunit.client.0.smithi102.stderr:noscrub is set 2024-08-19T16:05:20.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:492: TEST_just_deep_scrubs: ceph osd set nodeep-scrub 2024-08-19T16:05:20.414 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is set 2024-08-19T16:05:20.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:493: TEST_just_deep_scrubs: sleep 6 2024-08-19T16:05:26.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:494: TEST_just_deep_scrubs: date -Ins 2024-08-19T16:05:26.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:494: TEST_just_deep_scrubs: local now_is=2024-08-19T16:05:26,427479476+00:00 2024-08-19T16:05:26.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:495: TEST_just_deep_scrubs: declare -A sched_data 2024-08-19T16:05:26.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:496: TEST_just_deep_scrubs: local pgid=1.2 2024-08-19T16:05:26.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:499: TEST_just_deep_scrubs: set_query_debug 1.2 2024-08-19T16:05:26.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: local pgid=1.2 2024-08-19T16:05:26.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: ceph pg dump pgs_brief 2024-08-19T16:05:26.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: awk -v 'pg=^1.2' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-08-19T16:05:26.683 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs_brief 2024-08-19T16:05:26.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: local prim_osd=0 2024-08-19T16:05:26.694 INFO:tasks.workunit.client.0.smithi102.stdout:Setting scrub debug data. Primary for 1.2 is 0 2024-08-19T16:05:26.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: echo 'Setting scrub debug data. Primary for 1.2 is 0' 2024-08-19T16:05:26.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: get_asok_path osd.0 2024-08-19T16:05:26.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T16:05:26.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T16:05:26.696 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T16:05:26.696 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:05:26.696 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:05:26.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.536210/ceph-osd.0.asok 2024-08-19T16:05:26.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: CEPH_ARGS= 2024-08-19T16:05:26.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.536210/ceph-osd.0.asok scrubdebug 1.2 set sessions 2024-08-19T16:05:26.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:501: TEST_just_deep_scrubs: extract_published_sch 1.2 2024-08-19T16:05:26,427479476+00:00 2024-08-19T16:05:26,427479476+00:00 sched_data 2024-08-19T16:05:26.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.2 2024-08-19T16:05:26.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sched_data 2024-08-19T16:05:26.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2024-08-19T16:05:26,427479476+00:00 2024-08-19T16:05:26.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2024-08-19T16:05:26,427479476+00:00 2024-08-19T16:05:26.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=1 2024-08-19T16:05:26.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2024-08-19T16:05:26.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2024-08-19T16:05:27.031 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:05:27.149 INFO:tasks.workunit.client.0.smithi102.stdout:{"success":true}( 2024-08-19T16:05:27.150 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=22 2024-08-19T16:05:27.150 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=18 2024-08-19T16:05:27.150 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:05:27.150 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:05:27.150 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-20T16:05:11.571 (2024-08-20T16:05:11.571)' 2024-08-19T16:05:27.150 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=0 2024-08-19T16:05:27.150 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-19T16:05:11.571458+0000' 2024-08-19T16:05:27.150 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='0x0' 2024-08-19T16:05:27.150 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:05:27.150 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:05:27.150 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=8586240 2024-08-19T16:05:27.151 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:05:27.151 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:05:27.151 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=0 2024-08-19T16:05:27.151 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:05:27.151 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-20T16:05:11.571458+0000' 2024-08-19T16:05:27.151 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:05:27.151 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:05:27.151 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=22 2024-08-19T16:05:27.151 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=17 2024-08-19T16:05:27.151 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:05:27.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:502: TEST_just_deep_scrubs: local saved_last_stamp=2024-08-19T16:05:11.571458+0000 2024-08-19T16:05:27.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:503: TEST_just_deep_scrubs: local dbg_counter_at_start=8586240 2024-08-19T16:05:27.153 INFO:tasks.workunit.client.0.smithi102.stdout:test counter @ start: 8586240 2024-08-19T16:05:27.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:504: TEST_just_deep_scrubs: echo 'test counter @ start: 8586240' 2024-08-19T16:05:27.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:506: TEST_just_deep_scrubs: ceph tell 1.2 schedule-deep-scrub 2024-08-19T16:05:27.246 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-08-19T16:05:27.247 INFO:tasks.workunit.client.0.smithi102.stdout: "deep": true, 2024-08-19T16:05:27.247 INFO:tasks.workunit.client.0.smithi102.stdout: "must": false, 2024-08-19T16:05:27.247 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-08-05T16:03:47.246084+0000" 2024-08-19T16:05:27.247 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-08-19T16:05:27.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:508: TEST_just_deep_scrubs: sleep 5 2024-08-19T16:05:32.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:509: TEST_just_deep_scrubs: declare -A sc_data_2 2024-08-19T16:05:32.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:510: TEST_just_deep_scrubs: extract_published_sch 1.2 2024-08-19T16:05:26,427479476+00:00 2024-08-19T16:05:26,427479476+00:00 sc_data_2 2024-08-19T16:05:32.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.2 2024-08-19T16:05:32.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sc_data_2 2024-08-19T16:05:32.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2024-08-19T16:05:26,427479476+00:00 2024-08-19T16:05:32.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2024-08-19T16:05:26,427479476+00:00 2024-08-19T16:05:32.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=1 2024-08-19T16:05:32.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2024-08-19T16:05:32.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2024-08-19T16:05:32.531 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:05:32.649 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:05:32.649 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=22 2024-08-19T16:05:32.649 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=20 2024-08-19T16:05:32.649 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:05:32.649 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='deep scrub scheduled' 2024-08-19T16:05:32.649 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-19T16:05:33.011 (2024-08-06T16:03:47.246)' 2024-08-19T16:05:32.649 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=0 2024-08-19T16:05:32.649 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-05T16:03:47.246084+0000' 2024-08-19T16:05:32.650 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='0x0' 2024-08-19T16:05:32.650 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:05:32.650 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:05:32.650 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=8586240 2024-08-19T16:05:32.650 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:05:32.650 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:05:32.650 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=0 2024-08-19T16:05:32.650 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='queued for deep scrub' 2024-08-19T16:05:32.650 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='0' 2024-08-19T16:05:32.650 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=false 2024-08-19T16:05:32.650 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=false 2024-08-19T16:05:32.650 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=22 2024-08-19T16:05:32.650 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=19 2024-08-19T16:05:32.651 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:05:32.652 INFO:tasks.workunit.client.0.smithi102.stdout:test counter @ should show no change: 8586240 2024-08-19T16:05:32.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:511: TEST_just_deep_scrubs: echo 'test counter @ should show no change: ' 8586240 2024-08-19T16:05:32.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:512: TEST_just_deep_scrubs: (( 0 == 0 )) 2024-08-19T16:05:32.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:513: TEST_just_deep_scrubs: (( 8586240 == 8586240 )) 2024-08-19T16:05:32.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:516: TEST_just_deep_scrubs: ceph osd unset nodeep-scrub 2024-08-19T16:05:33.047 INFO:tasks.workunit.client.0.smithi102.stderr:nodeep-scrub is unset 2024-08-19T16:05:33.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:517: TEST_just_deep_scrubs: sleep 5 2024-08-19T16:05:38.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:518: TEST_just_deep_scrubs: expct_qry_duration=(['query_last_duration']='0' ['query_last_duration_neg']='not0') 2024-08-19T16:05:38.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:518: TEST_just_deep_scrubs: declare -A expct_qry_duration 2024-08-19T16:05:38.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:519: TEST_just_deep_scrubs: sc_data_2=() 2024-08-19T16:05:38.060 INFO:tasks.workunit.client.0.smithi102.stdout:test counter @ should be higher than before the unset: 2024-08-19T16:05:38.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:520: TEST_just_deep_scrubs: echo 'test counter @ should be higher than before the unset: ' 2024-08-19T16:05:38.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:521: TEST_just_deep_scrubs: wait_any_cond 1.2 10 2024-08-19T16:05:11.571458+0000 expct_qry_duration 'WaitingAfterScrub ' sc_data_2 2024-08-19T16:05:38.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.2 2024-08-19T16:05:38.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-08-19T16:05:38.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-08-19T16:05:11.571458+0000 2024-08-19T16:05:38.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=expct_qry_duration 2024-08-19T16:05:38.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sc_data_2 2024-08-19T16:05:38.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-08-19T16:05:38.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-08-19T16:05:38.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-08-19T16:05:38.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-08-19T16:05:38.063 INFO:tasks.workunit.client.0.smithi102.stdout:waiting for any condition (WaitingAfterScrub ): pg:1.2 dt:2024-08-19T16:05:11.571458+0000 (10 retries) 2024-08-19T16:05:38.829 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:05:38.946 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:05:38.947 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=23 2024-08-19T16:05:38.947 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=22 2024-08-19T16:05:38.947 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:05:38.947 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='deep scrub scheduled' 2024-08-19T16:05:38.947 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-19T16:05:38.994 (2024-08-06T16:03:47.246)' 2024-08-19T16:05:38.947 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=0 2024-08-19T16:05:38.947 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-05T16:03:47.246084+0000' 2024-08-19T16:05:38.947 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='0x0' 2024-08-19T16:05:38.947 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:05:38.947 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:05:38.947 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=8586240 2024-08-19T16:05:38.947 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:05:38.947 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:05:38.947 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=0 2024-08-19T16:05:38.948 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic deep scrub scheduled' 2024-08-19T16:05:38.948 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-19T16:05:38.994652+0000' 2024-08-19T16:05:38.948 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:05:38.948 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:05:38.948 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=23 2024-08-19T16:05:38.948 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=21 2024-08-19T16:05:38.948 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:05:38.950 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 1 ~ false / 22 / 21 / true / 2024-08-05T16:03:47.246084+0000 / deep scrub scheduled %%% query_last_duration_neg query_last_duration 2024-08-19T16:05:38.950 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-08-19T16:05:39.703 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:05:39.822 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:05:39.822 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=23 2024-08-19T16:05:39.822 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=30 2024-08-19T16:05:39.822 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:05:39.822 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:05:39.823 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-20T16:05:39.013 (2024-08-20T16:05:39.013)' 2024-08-19T16:05:39.823 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=0 2024-08-19T16:05:39.823 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-19T16:05:39.013575+0000' 2024-08-19T16:05:39.823 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='0x0' 2024-08-19T16:05:39.823 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:05:39.823 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:05:39.823 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=8586243 2024-08-19T16:05:39.823 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:05:39.823 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:05:39.824 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=0 2024-08-19T16:05:39.824 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic deep scrub scheduled' 2024-08-19T16:05:39.824 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-19T16:05:38.994652+0000' 2024-08-19T16:05:39.824 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:05:39.824 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:05:39.824 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=23 2024-08-19T16:05:39.824 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=21 2024-08-19T16:05:39.825 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:05:39.825 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 2 ~ false / 30 / 21 / true / 2024-08-19T16:05:39.013575+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-08-19T16:05:39.825 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-08-19T16:05:40.613 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:05:40.732 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:05:40.732 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=23 2024-08-19T16:05:40.732 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=30 2024-08-19T16:05:40.732 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:05:40.732 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:05:40.733 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-20T16:05:39.013 (2024-08-20T16:05:39.013)' 2024-08-19T16:05:40.733 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=0 2024-08-19T16:05:40.733 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-19T16:05:39.013575+0000' 2024-08-19T16:05:40.733 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='0x0' 2024-08-19T16:05:40.733 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:05:40.733 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:05:40.733 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=8586243 2024-08-19T16:05:40.733 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:05:40.733 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:05:40.733 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=0 2024-08-19T16:05:40.733 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic deep scrub scheduled' 2024-08-19T16:05:40.733 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-19T16:05:38.994652+0000' 2024-08-19T16:05:40.733 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:05:40.734 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:05:40.734 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=23 2024-08-19T16:05:40.734 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=21 2024-08-19T16:05:40.734 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:05:40.736 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 3 ~ false / 30 / 21 / true / 2024-08-19T16:05:39.013575+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-08-19T16:05:40.736 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-08-19T16:05:41.501 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:05:41.620 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:05:41.620 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=23 2024-08-19T16:05:41.620 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=30 2024-08-19T16:05:41.620 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:05:41.620 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:05:41.620 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-20T16:05:39.013 (2024-08-20T16:05:39.013)' 2024-08-19T16:05:41.620 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=0 2024-08-19T16:05:41.620 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-19T16:05:39.013575+0000' 2024-08-19T16:05:41.621 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='0x0' 2024-08-19T16:05:41.621 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:05:41.621 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:05:41.621 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=8586243 2024-08-19T16:05:41.621 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:05:41.621 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:05:41.621 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=0 2024-08-19T16:05:41.621 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic deep scrub scheduled' 2024-08-19T16:05:41.621 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-19T16:05:38.994652+0000' 2024-08-19T16:05:41.621 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:05:41.621 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:05:41.621 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=23 2024-08-19T16:05:41.621 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=21 2024-08-19T16:05:41.621 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:05:41.623 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 4 ~ false / 30 / 21 / true / 2024-08-19T16:05:39.013575+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-08-19T16:05:41.623 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-08-19T16:05:42.404 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:05:42.521 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:05:42.521 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=23 2024-08-19T16:05:42.521 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=30 2024-08-19T16:05:42.521 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:05:42.521 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:05:42.522 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-20T16:05:39.013 (2024-08-20T16:05:39.013)' 2024-08-19T16:05:42.522 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=0 2024-08-19T16:05:42.522 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-19T16:05:39.013575+0000' 2024-08-19T16:05:42.522 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='0x0' 2024-08-19T16:05:42.522 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:05:42.522 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:05:42.522 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=8586243 2024-08-19T16:05:42.522 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:05:42.522 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:05:42.522 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=0 2024-08-19T16:05:42.522 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:05:42.522 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-20T16:05:39.013575+0000' 2024-08-19T16:05:42.522 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:05:42.522 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:05:42.523 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=23 2024-08-19T16:05:42.523 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=29 2024-08-19T16:05:42.523 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:05:42.524 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 5 ~ false / 30 / 29 / true / 2024-08-19T16:05:39.013575+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-08-19T16:05:42.525 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-08-19T16:05:43.276 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:05:43.430 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:05:43.430 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=23 2024-08-19T16:05:43.430 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=30 2024-08-19T16:05:43.430 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:05:43.430 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:05:43.430 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-20T16:05:39.013 (2024-08-20T16:05:39.013)' 2024-08-19T16:05:43.430 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=0 2024-08-19T16:05:43.430 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-19T16:05:39.013575+0000' 2024-08-19T16:05:43.430 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='0x0' 2024-08-19T16:05:43.430 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:05:43.430 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:05:43.431 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=8586243 2024-08-19T16:05:43.431 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:05:43.431 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:05:43.431 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=0 2024-08-19T16:05:43.431 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:05:43.431 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-20T16:05:39.013575+0000' 2024-08-19T16:05:43.431 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:05:43.431 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:05:43.431 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=23 2024-08-19T16:05:43.431 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=29 2024-08-19T16:05:43.431 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:05:43.433 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 6 ~ false / 30 / 29 / true / 2024-08-19T16:05:39.013575+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-08-19T16:05:43.433 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-08-19T16:05:44.182 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:05:44.299 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:05:44.300 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=23 2024-08-19T16:05:44.300 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=30 2024-08-19T16:05:44.300 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:05:44.300 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:05:44.300 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-20T16:05:39.013 (2024-08-20T16:05:39.013)' 2024-08-19T16:05:44.300 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=0 2024-08-19T16:05:44.300 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-19T16:05:39.013575+0000' 2024-08-19T16:05:44.300 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='0x0' 2024-08-19T16:05:44.300 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:05:44.300 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:05:44.300 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=8586243 2024-08-19T16:05:44.300 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:05:44.300 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:05:44.301 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=0 2024-08-19T16:05:44.301 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:05:44.301 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-20T16:05:39.013575+0000' 2024-08-19T16:05:44.301 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:05:44.301 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:05:44.301 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=23 2024-08-19T16:05:44.301 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=29 2024-08-19T16:05:44.301 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:05:44.303 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 7 ~ false / 30 / 29 / true / 2024-08-19T16:05:39.013575+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-08-19T16:05:44.303 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-08-19T16:05:45.063 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:05:45.181 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:05:45.181 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=23 2024-08-19T16:05:45.181 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=30 2024-08-19T16:05:45.181 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:05:45.181 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:05:45.182 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-20T16:05:39.013 (2024-08-20T16:05:39.013)' 2024-08-19T16:05:45.182 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=0 2024-08-19T16:05:45.182 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-19T16:05:39.013575+0000' 2024-08-19T16:05:45.182 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='0x0' 2024-08-19T16:05:45.182 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:05:45.182 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:05:45.182 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=8586243 2024-08-19T16:05:45.182 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:05:45.182 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:05:45.182 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=0 2024-08-19T16:05:45.182 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:05:45.182 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-20T16:05:39.013575+0000' 2024-08-19T16:05:45.182 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:05:45.183 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:05:45.183 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=23 2024-08-19T16:05:45.183 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=29 2024-08-19T16:05:45.183 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:05:45.184 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 8 ~ false / 30 / 29 / true / 2024-08-19T16:05:39.013575+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-08-19T16:05:45.184 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-08-19T16:05:45.933 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:05:46.051 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:05:46.051 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=23 2024-08-19T16:05:46.051 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=30 2024-08-19T16:05:46.051 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:05:46.051 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:05:46.052 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-20T16:05:39.013 (2024-08-20T16:05:39.013)' 2024-08-19T16:05:46.052 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=0 2024-08-19T16:05:46.052 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-19T16:05:39.013575+0000' 2024-08-19T16:05:46.052 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='0x0' 2024-08-19T16:05:46.052 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:05:46.052 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:05:46.052 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=8586243 2024-08-19T16:05:46.052 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:05:46.052 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:05:46.052 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=0 2024-08-19T16:05:46.052 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:05:46.052 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-20T16:05:39.013575+0000' 2024-08-19T16:05:46.052 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:05:46.052 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:05:46.053 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=23 2024-08-19T16:05:46.053 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=29 2024-08-19T16:05:46.053 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:05:46.055 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 9 ~ false / 30 / 29 / true / 2024-08-19T16:05:39.013575+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-08-19T16:05:46.055 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-08-19T16:05:46.805 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-08-19T16:05:46.923 INFO:tasks.workunit.client.0.smithi102.stdout:( 2024-08-19T16:05:46.923 INFO:tasks.workunit.client.0.smithi102.stdout:[query_epoch]=23 2024-08-19T16:05:46.923 INFO:tasks.workunit.client.0.smithi102.stdout:[query_seq]=30 2024-08-19T16:05:46.923 INFO:tasks.workunit.client.0.smithi102.stdout:[query_active]=false 2024-08-19T16:05:46.923 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule]='scrub scheduled' 2024-08-19T16:05:46.923 INFO:tasks.workunit.client.0.smithi102.stdout:[query_schedule_at]='2024-08-20T16:05:39.013 (2024-08-20T16:05:39.013)' 2024-08-19T16:05:46.923 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_duration]=0 2024-08-19T16:05:46.923 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_stamp]='2024-08-19T16:05:39.013575+0000' 2024-08-19T16:05:46.923 INFO:tasks.workunit.client.0.smithi102.stdout:[query_last_scrub]='0x0' 2024-08-19T16:05:46.923 INFO:tasks.workunit.client.0.smithi102.stdout:[query_is_future]=true 2024-08-19T16:05:46.924 INFO:tasks.workunit.client.0.smithi102.stdout:[query_vs_date]=true 2024-08-19T16:05:46.924 INFO:tasks.workunit.client.0.smithi102.stdout:[query_scrub_seq]=8586243 2024-08-19T16:05:46.924 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_pg_state]='active+clean' 2024-08-19T16:05:46.924 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_state_has_scrubbing]=false 2024-08-19T16:05:46.924 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_last_duration]=0 2024-08-19T16:05:46.924 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-08-19T16:05:46.924 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_schedule_at]='2024-08-20T16:05:39.013575+0000' 2024-08-19T16:05:46.924 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_is_future]=true 2024-08-19T16:05:46.924 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_vs_date]=true 2024-08-19T16:05:46.924 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_reported_epoch]=23 2024-08-19T16:05:46.924 INFO:tasks.workunit.client.0.smithi102.stdout:[dmp_seq]=29 2024-08-19T16:05:46.924 INFO:tasks.workunit.client.0.smithi102.stdout:) 2024-08-19T16:05:46.926 INFO:tasks.workunit.client.0.smithi102.stdout:--> loop: 10 ~ false / 30 / 29 / true / 2024-08-19T16:05:39.013575+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-08-19T16:05:46.926 INFO:tasks.workunit.client.0.smithi102.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-08-19T16:05:46.926 INFO:tasks.workunit.client.0.smithi102.stdout:WaitingAfterScrub : wait_any_cond(): failure. Note: query-active=false 2024-08-19T16:05:46.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:156: wait_any_cond: return 1 2024-08-19T16:05:46.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:521: TEST_just_deep_scrubs: return 1 2024-08-19T16:05:46.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: return 1 2024-08-19T16:05:46.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2024-08-19T16:05:46.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-test 1 2024-08-19T16:05:46.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-08-19T16:05:46.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-08-19T16:05:46.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-08-19T16:05:46.928 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T16:05:46.928 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T16:05:46.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T16:05:46.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T16:05:46.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T16:05:47.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T16:05:47.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T16:05:47.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T16:05:47.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T16:05:47.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T16:05:47.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T16:05:47.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T16:05:47.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T16:05:47.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T16:05:47.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T16:05:47.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T16:05:47.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T16:05:47.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-08-19T16:05:47.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-08-19T16:05:47.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-08-19T16:05:47.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-scrub-test/mgr.x.log td/osd-scrub-test/mon.a.log td/osd-scrub-test/osd.0.log td/osd-scrub-test/osd.1.log td/osd-scrub-test/osd.2.log /home/ubuntu/cephtest/archive/log 2024-08-19T16:05:47.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-08-19T16:05:47.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T16:05:47.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T16:05:47.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.536210 2024-08-19T16:05:47.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.536210 2024-08-19T16:05:47.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T16:05:47.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T16:05:47.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T16:05:47.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 1 2024-08-19T16:05:47.054 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2024-08-19T16:05:47.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-08-19T16:05:47.338 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-test.sh) on smithi102 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=78215f5a1cf5eaf321d506a66539942e12851f34 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh' 2024-08-19T16:05:47.550 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6c08f1fdaab14545983e6ecaae789eea Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-test.sh) on smithi102 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=78215f5a1cf5eaf321d506a66539942e12851f34 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh' 2024-08-19T16:05:47.553 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-19T16:05:47.566 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-19T16:05:47.567 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-19T16:05:47.635 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-19T16:05:47.645 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-19T16:05:47.645 DEBUG:teuthology.orchestra.run.smithi102:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T16:05:47.659 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-08-19T16:05:47.663 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T16:05:47.663 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-08-19T16:05:47.663 INFO:teuthology.orchestra.run.smithi102.stdout:^* hv01.front.sepia.ceph.com 3 7 377 98 +521us[ +493us] +/- 54ms 2024-08-19T16:05:47.663 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 102 -956us[ -984us] +/- 62ms 2024-08-19T16:05:47.663 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +374us[ +374us] +/- 66ms 2024-08-19T16:05:47.663 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T16:05:47.664 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-19T16:05:47.675 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-19T16:05:47.675 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-19T16:05:47.697 DEBUG:teuthology.orchestra.run.smithi102:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-19T16:05:47.748 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-19T16:05:47.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T16:05:47.814 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-19T16:05:47.823 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-19T16:05:47.833 INFO:teuthology.task.internal:Duration was 7255.755525 seconds 2024-08-19T16:05:47.833 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-19T16:05:47.845 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-19T16:05:47.845 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-19T16:05:47.892 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T16:05:48.323 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-19T16:05:48.324 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi102.front.sepia.ceph.com 2024-08-19T16:05:48.324 DEBUG:teuthology.orchestra.run.smithi102:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-19T16:05:48.385 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-19T16:05:48.385 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-19T16:05:48.451 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-19T16:05:48.451 DEBUG:teuthology.orchestra.run.smithi102:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-19T16:05:48.555 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-19T16:05:48.565 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-19T16:05:48.565 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-19T16:05:48.589 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-19T16:05:48.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-19T16:05:48.652 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = core 2024-08-19T16:05:48.672 DEBUG:teuthology.orchestra.run.smithi102:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-19T16:05:48.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T16:05:48.727 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-19T16:05:48.736 INFO:teuthology.task.internal:Transferring archived files... 2024-08-19T16:05:48.738 DEBUG:teuthology.misc:Transferring archived files from smithi102:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-18_21:00:03-rados-squid-distro-default-smithi/7859720/remote/smithi102 2024-08-19T16:05:48.741 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-19T16:05:49.620 INFO:teuthology.task.internal:Removing archive directory... 2024-08-19T16:05:49.620 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-19T16:05:49.667 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-19T16:05:49.677 INFO:teuthology.task.internal:Not uploading archives. 2024-08-19T16:05:49.677 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-19T16:05:49.687 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-19T16:05:49.688 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-19T16:05:49.722 INFO:teuthology.orchestra.run.smithi102.stdout: 262218 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 19 16:05 /home/ubuntu/cephtest 2024-08-19T16:05:49.723 INFO:teuthology.orchestra.run.smithi102.stdout: 403366 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 19 14:08 /home/ubuntu/cephtest/mnt.0 2024-08-19T16:05:49.723 INFO:teuthology.orchestra.run.smithi102.stdout: 403367 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 19 15:51 /home/ubuntu/cephtest/mnt.0/client.0 2024-08-19T16:05:49.723 INFO:teuthology.orchestra.run.smithi102.stdout: 403411 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 19 16:05 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-19T16:05:49.723 INFO:teuthology.orchestra.run.smithi102.stdout: 415695 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 19 16:05 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-08-19T16:05:49.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T16:05:49.724 INFO:teuthology.orchestra.run.smithi102.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-08-19T16:05:49.725 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-19T16:05:49.731 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-19T16:05:49.745 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-19T16:05:49.756 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi102 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-19T16:05:49.760 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} duration: 7255.755525350571 failure_reason: 'Command failed (workunit test scrub/osd-scrub-test.sh) on smithi102 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=78215f5a1cf5eaf321d506a66539942e12851f34 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6c08f1fdaab14545983e6ecaae789eea status: fail success: false 2024-08-19T16:05:49.760 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-19T16:05:49.978 INFO:teuthology.run:FAIL